Deep learning Minichallenge 1: Hyperparameter tuning of CNN model¶

Weiping Zhang

FS2023

0 Challenge condition¶

  • Dataset: CIFAR10 I selected this dataset, because it contains a good level of complexity (10 classes, 3 color channels), at the same time it is also relatively small (low resolution (32x32 pixels),small datasize comparing to other deep learning datasets), this makes training models faster. I could train models very quick with different archtectures, optimizers, generalizations, learning rates, batch sizes.

  • Deep learning Model: 2D CNN 2D-CNN is designed specifically to process and analyze 2-dimensional data, such as images. This means it is particularly suitable for CIFAR10 image classification tasks due to its ability to effectively capture spatial patterns and hierarchies of features.

  • Deep learning Framework: PyTorch

  • MLOps Platform: wandb

1 Data understanding¶

1.1 load libraries and connect to wandb¶

In [1]:
# load libraries
import math
import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn
import torch.optim as optim
from torch.utils.data import DataLoader
import torchvision
import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torch.utils.data import Subset
from torchvision.transforms import ToTensor
import plotly.express as px
import wandb
api = wandb.Api()
wandb.login()
wandb: Currently logged in as: weiping-zhang (data_science2021). Use `wandb login --relogin` to force relogin
Out[1]:
True

1.2 load and normalize CIFAR train and test dataset¶

In [2]:
# Define transforms for data preprocessing
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4), # pad the image by 4 pixels padding=4, then randomly crop the image back to size 32x32
    transforms.RandomHorizontalFlip(), # randomly flip image horizontally, to augment data
    transforms.ToTensor(), # convert image to PyTorch tensor with shape (channels, height, width)
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) # normalize image values of each color channel to mean 0 and std 1
])
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

# Load the CIFAR-10 dataset
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
Files already downloaded and verified
Files already downloaded and verified

1.3 check if the data labels are balance¶

In [3]:
class_labels = trainset.classes

class_counts = torch.bincount(torch.tensor(trainset.targets))
class_counts_test = torch.bincount(torch.tensor(testset.targets))

plt.figure(figsize=(12, 3))
plt.bar(class_labels, class_counts)
plt.xlabel('Class Labels')
plt.ylabel('Number of Samples')
plt.title('Class Distribution (trainset)')
#plt.xticks(rotation=0)
plt.figure(figsize=(12, 3))
plt.bar(class_labels, class_counts_test)
plt.xlabel('Class Labels')
plt.ylabel('Number of Samples')
plt.title('Class Distribution (testset)')
plt.show()
  • both train and test sets have balanced classes

1.4 create small subset with one batch for quick test¶

In [4]:
# create a small train set with one batch and one small test set to quickly exam the model configurations
subset_indices = range(4)
trainset_small = Subset(trainset, subset_indices)
subset_indices = range(4)
testset_small = Subset(testset, subset_indices)
# small data loaders
trainloader_small = torch.utils.data.DataLoader(trainset_small, batch_size=4, shuffle=True, num_workers=2)
testloader_small = torch.utils.data.DataLoader(testset_small, batch_size=4, shuffle=False, num_workers=2)
In [5]:
# plot the pixel value distribution of the first images in the trainset_small set
for images, labels in trainloader_small:
    image = images[0]
    flattened_image = image.flatten()
    # plot
    plt.figure(figsize=(12, 3))
    plt.hist(flattened_image.numpy(), bins = 256, range=(-1.5, 1.5))
    plt.title('Pixel Value Distribution')
    plt.xlabel('Pixel Value')
    plt.ylabel('Frequency')
    plt.show()

1.5 Brief summary of data understanding¶

  • CIFAR10 dataset has 10 classes
  • Both train and test set have balanced classes
  • After normalization, the pixel values are in range [-1,1] as we could see from the upper plot.
  • The pixel values of train set are normal distributed.
  • Normalization is essential for deep learning models, because it will help to
      - improve convergence speed in training process and model stability.
      - improve generalization
      - prevent too large or too small gradients during backpropagation process.

2 Modeling and Evaluation¶

2.1 select deep learning models and define the architecture¶

  • choose 2D-Convolutional Neural Networks (CNNs) for CIFAR10 classification, because
    • they are specifically designed to capture spatial information from iamges
    • they can recognize patterns regardless of their location in the image
    • weights in kernel/filter will be used to across the whole image, this means reduced number of parameters compared to fully connected networks. This will make CNN more efficient and less prone to overfitting
    • they contain convolutional layers, pooling layers, and fully connected layers, all these together allow them to learn hierarchical feature extraction.
    • demonstrated good performance on image classification tasks (also include CIFAR10 datasets)
    • Overall, I think 2D CNNs is definitely a suitable choice for this challenge, and a good start for beginners to understand deep learning.
  • I will start with a simple 2D-CNN architecture with
    • 2 convolutional layers
      • no stride
      • no padding
    • 2 fully connected layers

2.2 define training models, which include the possibility to configure the following hyperparameters¶

  • data types
    • small dataset: to ensure the model is learning properly (if yes, the train set accuracy should be able to reach 100%)
    • normal dataset: the complete dataset
  • batch size
  • learning rate
  • convolutional hyperparameter: number of filter
  • optimizer
    • sgd
    • adam
  • regularization & strength
  • early stop: when train loss is not decreasing over n epochs, stop the training process

2.3 define the metric and criterion to evaluate models performance¶

  • evaluation metric: accurcy, as

    • it provides a straightforward interpretation of the model's performance: the proportion of predictions that the model got correct.
    • in this challenge, the dataset classes are balanced, all classes are equally important, so accuracy is a suiteable metric
    • it provides direct comparison between different models and approaches on the same dataset

    • train accuracy: is used to assess the model's ability to learn and fit trainset data

    • test accuracy: provides an estimate of the model's performance on unseen data. It is an indicator of the model generalization.
    • accuracy difference between train and test set: known as the "train-test" gap,it provides insights into the performance and generalization of a model. Foe example, a very large difference may indicate an overfitting, while a negative difference means underfitting.
  • optimization criterion: cross entropy loss, which meansures the discrepancy between the predicted output and the true target values during training. It is often to optimize model by minimizing the loss.

    • its function is smooth and is easier for optimization algorithms (like gradient descent) to find the minimum
    • it is also applicable for multi-class classification problems
  • all criterias will be logged in wandb
In [7]:
# Define the 2D CNN architecture
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        # 3 input channels, 32 output channels (32 filters), kernel size 3
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3) 
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3) 
        self.relu = nn.ReLU() # keep only positive values, negative values are replaced with 0
        self.pool = nn.MaxPool2d(2)
        # flattened the output into a vector 
        self.fc1 = nn.Linear(64 * 14 * 14, 128) 
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - Maxpool - fc1 - Relu - Dropout - fc2 - softmax
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
     
        # flatten the output into a vector
        x = x.view(-1, 64 * 14 * 14)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = nn.functional.softmax(self.fc2(x), dim=1) # softmax: probability distribution over 10 classes
        return x


def plot_results(train_losses, train_accuracies, test_accuracies, diff_accuracies):
    epochs = range(1, len(train_losses) + 1)
    plt.figure(figsize=(12, 4))
    plt.subplot(1, 2, 1)
    plt.plot(epochs, train_losses, label="Training Loss")
    plt.xlabel("Epochs")
    plt.ylabel("Loss")
    plt.legend()

    plt.subplot(1, 2, 2)
    plt.plot(epochs, train_accuracies, label="Training Accuracy")
    plt.plot(epochs, test_accuracies, label="Test Accuracy")
    plt.plot(epochs, diff_accuracies, label="Accuracy Difference")
    plt.xlabel("Epochs")
    plt.ylabel("Accuracy")
    plt.legend()
    plt.show()
    

def cnn_train_evaluation(config):
    wandb.init()
    config = wandb.config 
    
    if config['data_size'] == 'small':
        trainloader = trainloader_small
        testloader = testloader_small
    elif config['data_size'] == 'normal':
        trainloader = torch.utils.data.DataLoader(trainset, batch_size = config.batch_size, shuffle=True, num_workers=2)
        testloader = torch.utils.data.DataLoader(testset, batch_size = config.batch_size, shuffle=False, num_workers=2)

    net = Net(config)
    reg_type = config['regularization']
    reg_strength = config['regularization_strength']

    # Define the loss function, optimizer, regularization
    criterion = nn.CrossEntropyLoss()
    if config['optimizer'] == 'sgd':
        if reg_type == 'l1':
            optimizer = optim.SGD(net.parameters(), lr=config['learning_rate'], momentum=config['momentum'], weight_decay=reg_strength)
        elif reg_type == 'l2':
            optimizer = optim.SGD(net.parameters(), lr=config['learning_rate'], momentum=config['momentum'], weight_decay=reg_strength)
        else:
            optimizer = optim.SGD(net.parameters(), lr=config['learning_rate'], momentum=config['momentum'])
    elif config['optimizer']== 'adam':
        if reg_type == 'l1':
            optimizer = optim.Adam(net.parameters(), lr=config['learning_rate'], weight_decay=reg_strength)
        elif reg_type == 'l2':
            optimizer = optim.Adam(net.parameters(), lr=config['learning_rate'], weight_decay=reg_strength)
        else:
            optimizer = optim.Adam(net.parameters(), lr=config['learning_rate'])

    
    # append the loss, accuracy values in each epoch for plotting
    train_losses = []
    train_accuracies = []
    test_accuracies = []
    diff_accuracies = [] # Accuracy difference between train and testset

    # add early stopping: if the loss not decrease for a certain epochs, then stop the training process
    # initialize early stopping criterion
    early_stop_epochs = config['early_stop_epochs']
    early_stop_counter = 0
    best_loss = float('inf')
    
    # Train the model
    for epoch in range(config['epochs']):
        running_loss = 0.0
        total_train = 0 
        correct_train = 0 
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            optimizer.zero_grad()
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            if reg_type == 'l1':
                l1_reg = torch.tensor(0.)
                for param in net.parameters():
                    l1_reg += torch.norm(param, 1)
                loss += reg_strength * l1_reg
            elif reg_type == 'l2':
                l2_reg = torch.tensor(0.)
                for param in net.parameters():
                    l2_reg += torch.norm(param, 2)
                loss += reg_strength * l2_reg
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            _, predicted_train = torch.max(outputs.data, 1)
            total_train += labels.size(0)
            correct_train += (predicted_train == labels).sum().item()
        
        avg_loss = running_loss / len(trainloader)
        
        # check for early stopping
        if avg_loss < best_loss:
            early_stop_counter = 0
            best_loss = avg_loss
            torch.save(net.state_dict(), 'best_model.pt')
        else:
            early_stop_counter += 1
            if early_stop_counter >= early_stop_epochs:
                print('Early stopping')
                break
        
        train_accuracy = correct_train / total_train
        train_accuracies.append(train_accuracy)
        train_losses.append(avg_loss)
        wandb.log({"Training Loss": avg_loss}, step=epoch)
        wandb.log({"Training Accuracy": train_accuracy}, step=epoch)


        # Log test accuracy
        correct = 0
        total = 0
        with torch.no_grad():
            for data in testloader:
                images, labels = data
                outputs = net(images)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
        test_accuracy = correct / total
        wandb.log({"Test Accuracy": test_accuracy}, step=epoch)
        test_accuracies.append(test_accuracy)
        diff_accuracies.append(train_accuracy - test_accuracy)
        wandb.log({"Accuracy difference": (train_accuracy - test_accuracy)}, step=epoch)
    plot_results(train_losses, train_accuracies, test_accuracies, diff_accuracies)

2.4 test 2D-CNN model with small dataset¶

In [8]:
config = {
    "method": "grid",
    "metric": {'name': 'Test Accuracy', 'goal': 'maximize'},
    "parameters": {
        "optimizer": {"values": ["sgd"]},
        "data_size": {"values": ["small"]}, 
        "learning_rate": {"values": [0.01]},
        "momentum": {"values": [0]},    
        "epochs": {"values": [5000]},
        "early_stop_epochs": {"values": [2000]},  # set a large number to leave out early stop here
        "batch_size": {"values": [0]}, 
        "dropout_rate": {"values": [0]},     
        "regularization": {"values": ["none"]},  
        "regularization_strength": {"values": [0]} 
    }}
# Initialize the sweep
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-conv-layers_test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: pqbkzumq
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test/sweeps/pqbkzumq
wandb: Agent Starting Run: 2hasvv4m with config:
wandb: 	batch_size: 0
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_195604-2hasvv4m
Syncing run swift-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test/sweeps/pqbkzumq
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test/sweeps/pqbkzumq
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test/runs/2hasvv4m
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.020 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▁▅▅▅▅▅▅▅▅▅▅████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▅▅▅▅▅▅▅▅▅▅████████████████████████████
Training Loss█▆▄▄▄▄▄▄▄▄▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46127

View run swift-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs_test/runs/2hasvv4m
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_195604-2hasvv4m/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.

2.4.1 result interpretation¶

  • train on small data set reached accuracy of 100%, this means the model is able to learn, the model configuration could be used for further usage on normal dataset.
  • Now i would like to first start to compare which CNN architecture is most suitable for this dataset

3 Hyperparameter tuning¶

3.1 How many convolutional layers?¶

  • I plan to try the following four architectures

    • 2 conv layers + 2 fc layers (1 hidden layer, 1 output layer)
    • 0 conv layers + 3 fc layers (2 hidden, 1 output)
    • 1 conv layer + 2 fc layers
    • 3 conv layers + 2 fc layers
    • 4 conv layers + 2 fc layers
  • For each architechture it is necessary to first tune learning rate and batch size, because each architechture may have different optimal learning rate and batch size

  • In this section, I will keep the other hyperparameters as following (not change):

    • kernel size = 3
    • stride = 1
    • padding = 0
    • maxpooling with size = 2
    • optimizer: SGD (Stochastic Gradient Descent) is an commonly used optimization algorithm for training models in machine learning and deep learning trainings.

      • SGD aims to minimizes the loss function of a model by iteratively updating the parameters move a small step in the opposite direction of the gradients (steepest) of the loss function with respect to corresponding parameters.
      • The updating step size = gradient * learning rate.

        • the updated parameter = old parameter - gradient * learning rate
        • very small learning rate may lead to

          • slow convergence: because update step size is small, more iterations are needed to reach the expected performance, computational cost will be high.
          • get stuck in local optima: it can be hard with very small learning rate to escape local optima
        • very large learning rate may cause

          • overshooting: fail to converge
          • unstable training process: significant fluctuate between iterations
          • inaccurate convergence: parameters may be not fine-tuned, the found optima is not optimal enough.
      • In each epoch, SGD will randomly select a batch of the training data to compute the gradients and update the parameters to help preventing getting stuck in local optima.

        • very small batch size: fewer samples are used in one iteration, this may cause
          • noisy and less accurate gradient estimates
          • better generalization, as the batches are more diverse
          • larger computational cost for one epoch: as the parameters are more frequent updated within one epoch
          • slower convergence: as training with very small batch sizes may require more iterations to converge
        • very large batch size: more samples are used in one iteration, and may lead to
          • smoother optimization: as each update of parameter is based on more samples
          • faster training time: as modern hardwars has parallel processing capabilities
          • larger memory requirements
          • slower convergence: as the updates become less frequent within one epoch. The model may require more epochs to reach an optima.
      • strength and weakness of SGD:
        • strength (when use fine-tuned hyperparameters):
          • high computationally efficiency: as parameters are updated using only one batch at each iteration
          • fast convergence speed: as SGD updates parameters more frequent within one epoch, especially when training dataset is very large
          • high generalization: as randomly selecting batches introduces noise into training process
        • weakness (may happen when hyperparameter not well-tuned):
          • instability and slow convergence: due to the noisy updates by the random batch selecting
          • learning rate sensitivity
          • get stuck in local minima: especially in complex optimization landscapes. It may escape shallow local minima and find the global minima
          • not incorporate momentum: momentum helps accelerate convergence and smooth optimization process
        • Overall, SGD is a widely used efficient optimization algorithm, and it is able to handle large datasets. But it requires careful hyperparameter tuning.
    • no dropout
    • no regularization
    • early stop: when the loss not decrease for a certain epochs, stop the training process
      • use 20 early stop epochs for the normal dataset
      • use larger values for the small dataset

3.1.1 with 2 conv layers + 2 fc layers (1 hidden layer, 1 output layer)¶

3.1.1.1 tune learning rate & batch size¶
  • learning rate: 0.1, 0.01, 0.001
  • batch size: 16, 32, 64, 128
In [ ]:
config["parameters"]["data_size"]["values"] = ["normal"]
config["parameters"]["batch_size"]["values"] = [128,64,32,16]
config["parameters"]["learning_rate"]["values"] = [0.1,0.01,0.001]
config["parameters"]["epochs"]["values"] = [300]
config["parameters"]["early_stop_epochs"]["values"] = [20]
# Initialize the sweep
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-lr-bs", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 66fs2rmd
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
wandb: Agent Starting Run: bw4ki9ji with config:
wandb: 	batch_size: 128
wandb: 	dropout: 0.01
wandb: 	epochs: 300
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
wandb version 0.14.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.14.1
Run data is saved locally in /home/jovyan/work/wandb/run-20230412_151223-bw4ki9ji
Syncing run graceful-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/bw4ki9ji
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.007 MB of 0.023 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.299410…

Run history:


Estimation error▁▁▂▂▁▃▃▃▃▃▄▄▅▄▆▄▅▆▅▅▅▆▅▅▆▆▇▇▇▇▆▇▇█▇▇▇▇█▇
Test Accuracy▁▃▄▅▆▆▇▇▇▇▇▇▇▇▇█▇▇██████████████████████
Training Accuracy▁▂▄▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█▇█████████████████
Training Loss█▇▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Estimation error0.02536
Test Accuracy0.7745
Training Accuracy0.79986
Training Loss1.66075

View run graceful-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/bw4ki9ji
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230412_151223-bw4ki9ji/logs
wandb: Agent Starting Run: mtjy7t6u with config:
wandb: 	batch_size: 64
wandb: 	dropout: 0.01
wandb: 	epochs: 300
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
wandb version 0.14.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.14.1
Run data is saved locally in /home/jovyan/work/wandb/run-20230412_211417-mtjy7t6u
Syncing run breezy-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/mtjy7t6u
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.007 MB of 0.007 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Estimation error▁▃▃▄▅▄▅▃▄▄▄▄▅▅▅▆▄▅▄▄▄▄▅▅▅▄▅█▅▅▅▅▅▆▅▄▅▅▆▆
Test Accuracy▁▃▄▅▅▆▆▇▇▇▇▇▇▇▇▇███████████▇█████▇██████
Training Accuracy▁▃▄▅▆▆▆▇▇▇▇▇▇▇▇█████████████████████████
Training Loss█▆▅▄▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Estimation error0.00996
Test Accuracy0.7094
Training Accuracy0.71936
Training Loss1.74057

View run breezy-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/mtjy7t6u
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230412_211417-mtjy7t6u/logs
wandb: Agent Starting Run: 1qkkafx9 with config:
wandb: 	batch_size: 32
wandb: 	dropout: 0.01
wandb: 	epochs: 300
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
wandb version 0.14.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.14.1
Run data is saved locally in /home/jovyan/work/wandb/run-20230413_020013-1qkkafx9
Syncing run silvery-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/1qkkafx9
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.007 MB of 0.007 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Estimation error▁▄▃▆▃▄▄▄▄▃▅▄▅▅▇▄▅▅▅▄▆▅▅▇▅▅▅▅▅▄▇▆▄▅▅▆█▃▃▆
Test Accuracy▁▂▃▃▅▅▅▅▆▇▆▇▇▇▇▇███████▇██████▇▇▇▇▇▆▆▇▇▅
Training Accuracy▁▃▄▄▄▅▅▆▆▇▇▇▇▇▇▇████████████████▇▇▇▇▇▇▇▆
Training Loss█▆▆▅▅▄▄▄▃▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▂▃

Run summary:


Estimation error-0.02102
Test Accuracy0.5306
Training Accuracy0.50958
Training Loss1.95105

View run silvery-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/1qkkafx9
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230413_020013-1qkkafx9/logs
wandb: Agent Starting Run: 409sbjmv with config:
wandb: 	batch_size: 16
wandb: 	dropout: 0.01
wandb: 	epochs: 300
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
wandb version 0.14.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.14.1
Run data is saved locally in /home/jovyan/work/wandb/run-20230413_053844-409sbjmv
Syncing run giddy-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/66fs2rmd
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/409sbjmv
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.007 MB of 0.007 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Estimation error▁▄▅█▃▄▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅
Test Accuracy█▇▆▂▃▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▆█▇▅▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss▂▁▂▄▇▇▇██████████████

Run summary:


Estimation error0.0
Test Accuracy0.1
Training Accuracy0.1
Training Loss2.36115

View run giddy-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/409sbjmv
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230413_053844-409sbjmv/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.1.1.2 result of tuning (linked from wandb)¶
3.1.1.3 interpretation¶
  • For this 2 convolutional layer CNN model, learning_rate 0.01, batch_size 64 are most suitable, because
    • with learning_rate 0.001, models converge too slow ( need more epochs to reach comparable performance)
    • with learning_rate 0.1, 3 models failed to converge
    • therefore learning_rate 0.01 is optimal
    • with small batch_size, the test accuracy is increased, but the training time is also larger.
    • with batch_size of 64, model has the best balance between computing time and test accuracy

In [6]:
%wandb weiping-zhang/mc1-sgd-early-stop-lr-bs/reports/3-1-1-1-tune-learning-rate-batch-size--Vmlldzo0NDc0Njg5 -h 1024
3.1.1.4 use 500 epochs with optimal learning rate & batch size¶
In [ ]:
# use 500 epochs of 2 convolutional layers
# update learning_rate to 0.01, batch_size to 64
config["parameters"]["learning_rate"]["values"] = [0.01]
config["parameters"]["batch_size"]["values"] = [64]
config["parameters"]["epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv_layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
In [7]:
%wandb weiping-zhang/mc1-sgd-early-stop-n_conv_layers/reports/2-convolutional-layers--Vmlldzo0NDg0ODUy -h 1024

3.1.2 with 0 conv layers + 3 fc layers¶

3.1.2.1 re-define the 2D CNN architecture¶
In [10]:
# use 0 convolutional layers, 3 fully connected layers 
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3 * 32 * 32, 256) 
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # fc1 - Relu - Dropout - fc2 - Relu - Dropout - fc3 - softmax
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.relu(self.fc2(x))
        x = self.dropout(x)
        x = nn.functional.softmax(self.fc3(x), dim=1) 
        return x
3.1.2.2 test on small dataset to check if the new model is able to learn¶
In [8]:
config["parameters"]["data_size"]["values"] = ['small']
config["parameters"]["epochs"]["values"] = [2000]
config["parameters"]["early_stop_epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-conv-layers_test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
wandb: WARNING Calling wandb.login() after wandb.init() has no effect.
Create sweep with ID: qozzlbto
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
wandb: Waiting for W&B process to finish... (success).
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_212640-do0ckhtf
Syncing run trim-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/y1vtx3ct
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/y1vtx3ct
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/do0ckhtf
wandb: 🚀 View run trim-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/do0ckhtf
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230427_212640-do0ckhtf/logs
wandb: Agent Starting Run: r3xgc6x8 with config:
wandb: 	batch_size: 128
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_212657-r3xgc6x8
Syncing run stellar-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/r3xgc6x8
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.222829…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▅▁▅▅▅▅▅▅▅▅▅▅▅▅████▅██████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▅▁▅▅▅▅▅▅▅▅▅▅▅▅████▅██████
Training Loss████████▇▇▇▇▆▆▅▄▅▄▄▃▃▃▃▃▃▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.5182

View run stellar-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/r3xgc6x8
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_212657-r3xgc6x8/logs
wandb: Agent Starting Run: ygr81q7e with config:
wandb: 	batch_size: 128
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_215248-ygr81q7e
Syncing run driven-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ygr81q7e
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▁▁▁█████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁█████████████████████████████████████
Training Loss█▆▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46168

View run driven-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ygr81q7e
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_215248-ygr81q7e/logs
wandb: Agent Starting Run: yr2uo80q with config:
wandb: 	batch_size: 128
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_221419-yr2uo80q
Syncing run dazzling-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/yr2uo80q
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185482…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss▄█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46115

View run dazzling-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/yr2uo80q
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_221419-yr2uo80q/logs
wandb: Agent Starting Run: ymp7pna5 with config:
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_223615-ymp7pna5
Syncing run cerulean-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ymp7pna5
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185427…

Run history:


Accuracy difference▁▄▅▅▅█▅▇▇▅▇▇▅▅▅▇▇▇▇▇▇▇▇▇▇▇██████████████
Test Accuracy█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▃▅▅▅█▅▆▆▅▆▆▅▅▅▆▆▆▆▆▆▆▆▆▆▆██████████████
Training Loss█████████▇▇▆▆▆▅▅▄▄▃▃▃▃▃▂▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.49739

View run cerulean-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ymp7pna5
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_223615-ymp7pna5/logs
wandb: Agent Starting Run: uw0zzpji with config:
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_225829-uw0zzpji
Syncing run pleasant-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/uw0zzpji
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.222720…

Run history:


Accuracy difference▁▁▁█████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁█████████████████████████████████████
Training Loss██▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46132

View run pleasant-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/uw0zzpji
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_225829-uw0zzpji/logs
wandb: Agent Starting Run: hwmkx1av with config:
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_232116-hwmkx1av
Syncing run revived-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/hwmkx1av
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.431442…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46117

View run revived-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/hwmkx1av
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_232116-hwmkx1av/logs
wandb: Agent Starting Run: 1j1ko1uj with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230427_234406-1j1ko1uj
Syncing run pious-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/1j1ko1uj
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.020 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▁▁▂▄▅▅▅▅▅▇▅▅▅▅▅▅▅▅▅▅▇▇▇▇▇▇▇▇▇▇▇▇▇██████
Test Accuracy█████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▃▅▅▅▅▅▅▆▅▅▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆██████
Training Loss█████████▇▇▇▆▆▅▅▄▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▁▂▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.55514

View run pious-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/1j1ko1uj
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230427_234406-1j1ko1uj/logs
wandb: Agent Starting Run: w174y19v with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_000711-w174y19v
Syncing run daily-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/w174y19v
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185508…

Run history:


Accuracy difference▁▁▅█████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▅█████████████████████████████████████
Training Loss█▇▄▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46129

View run daily-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/w174y19v
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_000711-w174y19v/logs
wandb: Agent Starting Run: qhoz5cv6 with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_003019-qhoz5cv6
Syncing run wobbly-sweep-9 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/qhoz5cv6
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185491…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss█▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46116

View run wobbly-sweep-9 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/qhoz5cv6
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_003019-qhoz5cv6/logs
wandb: Agent Starting Run: 3431g2wn with config:
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_005347-3431g2wn
Syncing run dulcet-sweep-10 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/3431g2wn
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185444…

Run history:


Accuracy difference▁▁▃▃▆▆█▆▆▆▆▆▆▆▆▃▆▆▆▆▆▆▆▆▆███▆▆██████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▃▃▆▆█▆▆▆▆▆▆▆▆▃▆▆▆▆▆▆▆▆▆███▆▆██████████
Training Loss██████████▇▇▇▇▅▅▄▃▄▃▃▃▃▃▃▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.51169

View run dulcet-sweep-10 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/3431g2wn
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_005347-3431g2wn/logs
wandb: Agent Starting Run: ce5vjgc3 with config:
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_011743-ce5vjgc3
Syncing run hardy-sweep-11 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ce5vjgc3
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185444…

Run history:


Accuracy difference█▁▁█████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy█▁▁█████████████████████████████████████
Training Loss█▇▄▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46136

View run hardy-sweep-11 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/ce5vjgc3
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_011743-ce5vjgc3/logs
wandb: Agent Starting Run: d0mlgm14 with config:
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_014125-d0mlgm14
Syncing run playful-sweep-12 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/qozzlbto
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/d0mlgm14
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.421236…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46116

View run playful-sweep-12 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/d0mlgm14
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_014125-d0mlgm14/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
Error in callback <function _WandbInit._pause_backend at 0x7f478dce9bd0> (for post_run_cell):
---------------------------------------------------------------------------
BrokenPipeError                           Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/backcall/backcall.py:104, in callback_prototype.<locals>.adapt.<locals>.adapted(*args, **kwargs)
    102                 kwargs.pop(name)
    103 #            print(args, kwargs, unmatched_pos, cut_positional, unmatched_kw)
--> 104             return callback(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py:419, in _WandbInit._pause_backend(self)
    417 if self.backend.interface is not None:
    418     logger.info("pausing backend")  # type: ignore
--> 419     self.backend.interface.publish_pause()

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/interface/interface.py:665, in InterfaceBase.publish_pause(self)
    663 def publish_pause(self) -> None:
    664     pause = pb.PauseRequest()
--> 665     self._publish_pause(pause)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py:340, in InterfaceShared._publish_pause(self, pause)
    338 def _publish_pause(self, pause: pb.PauseRequest) -> None:
    339     rec = self._make_request(pause=pause)
--> 340     self._publish(rec)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py:51, in InterfaceSock._publish(self, record, local)
     49 def _publish(self, record: "pb.Record", local: Optional[bool] = None) -> None:
     50     self._assign(record)
---> 51     self._sock_client.send_record_publish(record)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py:221, in SockClient.send_record_publish(self, record)
    219 server_req = spb.ServerRequest()
    220 server_req.record_publish.CopyFrom(record)
--> 221 self.send_server_request(server_req)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py:155, in SockClient.send_server_request(self, msg)
    154 def send_server_request(self, msg: Any) -> None:
--> 155     self._send_message(msg)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py:152, in SockClient._send_message(self, msg)
    150 header = struct.pack("<BI", ord("W"), raw_size)
    151 with self._lock:
--> 152     self._sendall_with_error_handle(header + data)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py:130, in SockClient._sendall_with_error_handle(self, data)
    128 start_time = time.monotonic()
    129 try:
--> 130     sent = self._sock.send(data)
    131     # sent equal to 0 indicates a closed socket
    132     if sent == 0:

BrokenPipeError: [Errno 32] Broken pipe
3.1.2.3 tune learning rate & batch size on normal dataset¶
  • train accuracy on small dataset reached 100% , indicated the re-defined model is able to learn, could be applied on normal dataset
  • find the optimal learning rate and batch size of this model architechture
    • learning rate: 0.1, 0.01, 0.001
    • batch size: 16, 32, 64, 128
    • use 50 epochs to save computation time: as in section 3.1.1.2 showed that training with 50 epochs is already helpful to select optimal learning rate and batch size
In [ ]:
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["epochs"]["values"] = [50]
config["parameters"]["learning_rate"]["values"] = [0.001,0.01,0.1]
config["parameters"]["batch_size"]["values"] = [128,64,32,16]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-lr-bs", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
3.1.2.4 result of tuning (linked from wandb)¶
In [12]:
entity, project, sweep_id = "weiping-zhang","mc1-sgd-early-stop-lr-bs", "jqd8f38v" 
sweep = api.sweep(f"{entity}/{project}/{sweep_id}")
sweep.display(height=1024)
Out[12]:
True
3.1.2.5 interpretation¶
  • model with 0 convolutional layers + 3 fully connected layers, optimal learning rate is 0.01, optimal batch size is 32, because
    • learning rate 0.1 is too large, two of the four models failed to converge, large influctuation in test accuracy
    • learning rate 0.001 is too small, it is very slow to reach good performance as the other learning rates
    • so I select learning_rate 0.01
    • with the four batch_size options, the test accuracy increased with smaller batch size, but the training time is also larger.
    • with batch_size of 32, model has the best balance between computing time and test accuracy
    • therefore optimal batch size of this architecture is 32
3.1.2.6 use 500 epochs with optimal learning rate & batch size¶
In [ ]:
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["learning_rate"]["values"] = [0.01]
config["parameters"]["batch_size"]["values"] = [32]
config["parameters"]["epochs"]["values"] = [500]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv_layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: wzt3e13v
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/wzt3e13v
wandb: Agent Starting Run: 7qz1k8du with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 500
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_105103-7qz1k8du
Syncing run sunny-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/wzt3e13v
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/wzt3e13v
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/7qz1k8du
wandb: Network error (ConnectionError), entering retry loop.
In [14]:
sweep = api.sweep(f"weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/wzt3e13v")
sweep.display(height=1024)
Out[14]:
True

3.1.3 with 1 conv layer + 2 fc layers¶

3.1.3.1 re-define 2D CNN architecture¶
In [15]:
# use 1 convolutional layers
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(32 * 15 * 15, 128) 
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - Maxpool - fc1 - Relu - Dropout - fc2 - softmax
        x = self.pool(self.relu(self.conv1(x)))
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = nn.functional.softmax(self.fc2(x), dim=1) 
        return x
3.1.3.2 test on small dataset¶
In [7]:
config["parameters"]["data_size"]["values"] = ['small']
config["parameters"]["epochs"]["values"] = [2000]
config["parameters"]["early_stop_epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-conv-layers_test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 9plpr0j2
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/9plpr0j2
wandb: Agent Starting Run: lb2idy4e with config:
wandb: 	batch_size: 0
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 3000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
wandb: Currently logged in as: weiping-zhang. Use `wandb login --relogin` to force relogin
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_181734-lb2idy4e
Syncing run spring-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/9plpr0j2
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/9plpr0j2
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/lb2idy4e
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.020 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▅██████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▅██████████████████████████████████████
Training Loss█▄▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46143

View run spring-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/lb2idy4e
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_181734-lb2idy4e/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.1.3.3 tune learning rate & batch size on normal dataset for this model architecture¶
  • train accuracy on small dataset reached 100% , indicated the re-defined model is able to learn, could be applied on normal dataset
  • find the optimal learning rate and batch size of this model architechture
    • learning rate: 0.1, 0.01, 0.001
    • batch size: 16, 32, 64, 128
    • use 50 epochs to save computation time: as section 3.1.1.2 showed that training with 50 epochs is already helpful to select optimal learning rate and batch size
In [8]:
# tune learning_rate & batch_size
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["epochs"]["values"] = [50]
config["parameters"]["learning_rate"]["values"] = [0.001,0.01,0.1]
config["parameters"]["batch_size"]["values"] = [128,64,32,16]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-lr-bs", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: soly2oso
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
wandb: Agent Starting Run: va08qpgf with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_183306-va08qpgf
Syncing run sage-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/va08qpgf
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185756…

Run history:


Accuracy difference█▆▄▂▂▂▁▂▁▁▁▁▁▁▁▁▂▃▃▃▄▅▅▅▅▅▅▅▅▅▄▄▄▄▄▄▃▃▂▃
Test Accuracy▁▂▄▅▆▆▆▆▆▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇████
Training Accuracy▁▂▃▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇███
Training Loss████████▇▇▇▇▇▇▇▆▆▆▆▆▅▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁

Run summary:


Accuracy difference-0.01616
Test Accuracy0.2814
Training Accuracy0.26524
Training Loss2.20209

View run sage-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/va08qpgf
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_183306-va08qpgf/logs
wandb: Agent Starting Run: 5cjfro95 with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016669166183661824, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_185536-5cjfro95
Syncing run swift-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/5cjfro95
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.421035…

Run history:


Accuracy difference▁██▇▅▆▇▆▇▇█▅▇▆▆▇▆▆▆▇▆▇▆▇▇▇▆▆▆▆▇▅▆▆▇▆▇▆▆▆
Test Accuracy▁▁▁▂▃▃▃▄▄▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇█████
Training Accuracy▁▂▂▂▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████
Training Loss██▇▆▆▅▅▅▅▄▄▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02736
Test Accuracy0.4825
Training Accuracy0.45514
Training Loss2.01132

View run swift-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/5cjfro95
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_185536-5cjfro95/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: jj641jd2 with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_191801-jj641jd2
Syncing run spring-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/jj641jd2
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185791…

Run history:


Accuracy difference▁▃▅▅▆▅▅▄▆▄▆▄▅▅▅▅▆▆▅▅▆▅▅▅▇▆▆▆▆▅▅▆▆▅▅▆█▇▆▇
Test Accuracy▁▂▃▃▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Accuracy▁▃▃▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Loss█▇▆▅▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0175
Test Accuracy0.6522
Training Accuracy0.6347
Training Loss1.82625

View run spring-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/jj641jd2
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_191801-jj641jd2/logs
wandb: Agent Starting Run: igk84odl with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_193932-igk84odl
Syncing run dainty-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/igk84odl
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.008 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.392640…

Run history:


Accuracy difference▅▅▁▁▂▃▅▆█▇▇▇▇▇▇████▆▅▅▅▅▃▃▂▄▃▅▃▄▅▄▄▅▅▄▄▅
Test Accuracy▁▂▃▄▄▄▄▄▄▄▄▄▄▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇█████████
Training Accuracy▁▂▂▃▄▄▄▄▄▄▄▄▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss██████▇▇▇▇▆▆▅▅▅▅▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01512
Test Accuracy0.3167
Training Accuracy0.30158
Training Loss2.15938

View run dainty-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/igk84odl
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_193932-igk84odl/logs
wandb: Agent Starting Run: ics4z9iv with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_200642-ics4z9iv
Syncing run super-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ics4z9iv
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185756…

Run history:


Accuracy difference▁▂▄▂▄▄▅▆▄▅▇▇▇▅▄▄▅▃▅▇▇▆▄▃▃▅▄▇▅▅▅▆█▆▅▆▆▆▅▇
Test Accuracy▁▂▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇███▇███████
Training Accuracy▁▂▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02042
Test Accuracy0.5618
Training Accuracy0.54138
Training Loss1.92493

View run super-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ics4z9iv
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_200642-ics4z9iv/logs
wandb: Agent Starting Run: kd6os70a with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_203513-kd6os70a
Syncing run trim-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/kd6os70a
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▁▄▄▂▃▂▄▃▃▄▄▅█▄▃▅▃▃▄▃▃▃▃▃▄▃▄▃▃▅▆▄▄▄▄▅▃▃▄▃
Test Accuracy▁▂▃▄▅▅▅▅▆▆▆▆▄▆▇▆▇▇▇▇█▇██▇█▇██▇▇████▇████
Training Accuracy▁▃▄▄▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇█▇█████████████████
Training Loss█▆▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02522
Test Accuracy0.6629
Training Accuracy0.63768
Training Loss1.82136

View run trim-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/kd6os70a
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_203513-kd6os70a/logs
wandb: Agent Starting Run: pvb5xxmc with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_210302-pvb5xxmc
Syncing run trim-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/pvb5xxmc
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.223036…

Run history:


Accuracy difference▆▃▁▄█▇▄▇▆▆▄▄▄▅▅▄▅▃▃▃▄▅▅▅▅▅▅▆▆▄▅▆▄▄▆▆▄▄▅▆
Test Accuracy▁▂▂▃▃▃▄▄▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇████████
Training Accuracy▁▁▂▂▃▃▄▄▄▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇██████
Training Loss████▇▇▆▆▅▅▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01632
Test Accuracy0.3778
Training Accuracy0.36148
Training Loss2.10262

View run trim-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/pvb5xxmc
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_210302-pvb5xxmc/logs
wandb: Agent Starting Run: 96uznue9 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_213857-96uznue9
Syncing run happy-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/96uznue9
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.005 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.833612…

Run history:


Accuracy difference▁▅▇▇▆▆▇▇▆▆▆█▆▆▇▇█▆▇▇▇▇▇▇▇▇▇▇▇██▇▆▇▆▇▇█▇█
Test Accuracy▁▂▃▃▄▄▄▄▅▅▆▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▆▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02178
Test Accuracy0.6183
Training Accuracy0.59652
Training Loss1.8681

View run happy-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/96uznue9
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_213857-96uznue9/logs
wandb: Agent Starting Run: z0zy6gc0 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_221554-z0zy6gc0
Syncing run dauntless-sweep-9 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/z0zy6gc0
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▅▇▇▅▆▆▅▇▆▆▆▇▆▅▇█▆▇▆▇█▇▇▇▆█▆▅█▆▇▇▇▅▅▆▆
Test Accuracy▁▃▃▄▆▆▆▇▆▇▇█▇██▇▇█▇██▇▇▇▇█▇██▇█▇▇▇██▇█
Training Accuracy▁▄▅▆▆▆▇▇▇▇████████████████████████████
Training Loss█▅▄▃▃▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.04678
Test Accuracy0.575
Training Accuracy0.52822
Training Loss1.93162

View run dauntless-sweep-9 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/z0zy6gc0
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_221554-z0zy6gc0/logs
wandb: Agent Starting Run: bl1js2ru with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_224559-bl1js2ru
Syncing run peachy-sweep-10 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/bl1js2ru
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▁▆▆█▇▆▆▅▅▄▄▅▅▅▆▆▆▆▆▆▆▆▇▆▆▆▇▆█▆█▇▇▇▇▇▇▇▆▆
Test Accuracy▁▁▁▂▃▃▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇████
Training Accuracy▁▂▂▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss██▇▇▆▅▅▅▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02258
Test Accuracy0.4506
Training Accuracy0.42802
Training Loss2.03677

View run peachy-sweep-10 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/bl1js2ru
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_224559-bl1js2ru/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: wgqel0sv with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230428_233945-wgqel0sv
Syncing run young-sweep-11 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/wgqel0sv
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185774…

Run history:


Accuracy difference▁▄▅▅▇▄▅▅▆▄▅▅▅▅▆▆▆▇▆▆▆█▆▆▆▆▆▇▆▇▆▇█▇▆▇▇▇▇▇
Test Accuracy▁▂▃▃▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇█▇████████████
Training Accuracy▁▂▃▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Loss█▇▆▅▅▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02126
Test Accuracy0.6332
Training Accuracy0.61194
Training Loss1.85013

View run young-sweep-11 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/wgqel0sv
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230428_233945-wgqel0sv/logs
wandb: Agent Starting Run: a7eol2j6 with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 50
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_003049-a7eol2j6
Syncing run jumping-sweep-12 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/soly2oso
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/a7eol2j6
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▃▅▂▄▄█▂▇▆▂▅▄▄▄▆▁▁▆▁▅▅▄▅
Test Accuracy▄▆▆█▇▇▅▇▅▅▆▅▅▅▄▃▄▄▁▃▁▂▂▂
Training Accuracy▂▆▇████▇▇▇▆▆▅▄▄▄▂▃▂▁▁▂▂▃
Training Loss▆▃▂▁▁▁▁▂▂▂▃▃▄▅▅▅▇▆▇██▇▇▆

Run summary:


Accuracy difference-0.02444
Test Accuracy0.3556
Training Accuracy0.33116
Training Loss2.12991

View run jumping-sweep-12 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/a7eol2j6
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_003049-a7eol2j6/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.1.3.4 result of tuning (linked from wandb)¶
In [30]:
project, sweep_id = "mc1-sgd-early-stop-lr-bs", "soly2oso" 
sweep = api.sweep(f"{entity}/{project}/{sweep_id}")
sweep.display(height=500)
Out[30]:
True
3.1.3.5 interpretation¶
  • model with 1 convolutional layers + 2 fully connected layers, optimal learning rate is 0.01, optimal batch size is 32, because
    • learning rate 0.1 is too large, two of the four models failed to converge, large influctuation in test accuracy
    • learning rate 0.001 is too small, it is very slow to reach good performance as the other learning rates
    • so I select learning_rate 0.01
    • with the four batch_size options, the test accuracy increased with smaller batch size, but the training time is also larger.
    • with batch_size of 32, model has the best balance between computing time and test accuracy
    • therefore optimal batch size of this architecture is 32
3.1.2.6 use 500 epochs with optimal learning rate & batch size¶
In [9]:
# update learning rate & batch size, train on normal dataset
config["parameters"]["learning_rate"]["values"] = [0.01]
config["parameters"]["batch_size"]["values"] = [32]
config["parameters"]["epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv_layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: ssah2iar
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/ssah2iar
wandb: Agent Starting Run: 7ih1lrmb with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 500
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_005752-7ih1lrmb
Syncing run driven-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/ssah2iar
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/ssah2iar
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/7ih1lrmb
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185748…

Run history:


Accuracy difference▂▁▁▃▁▂▂▃▃▃▄▄▄▄▄▆▄▆▅▅▅▆▆▆▇▆▇▆▆▆▆▇▆█▇▇▇█▇▇
Test Accuracy▁▄▅▅▆▆▇▇▇▇▇▇▇▇▇▇█▇██████████████████████
Training Accuracy▁▃▄▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Loss█▆▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.02376
Test Accuracy0.7448
Training Accuracy0.76856
Training Loss1.69334

View run driven-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/7ih1lrmb
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_005752-7ih1lrmb/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [16]:
sweep = api.sweep(f"weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/ssah2iar")
sweep.display(height=1024)
Out[16]:
True

3.1.4 with 3 conv layers + 2 fc layers¶

3.1.4.1 re-define 2D CNN architecture¶
In [17]:
# use 3 convolutional layers
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(3, 32, kernel_size=3) 
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3) 
        self.conv3 = nn.Conv2d(64,256,kernel_size=3) 
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        
        self.fc1 = nn.Linear(256 * 6 * 6, 128) 
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - Maxpool - CONV3 - Relu - Maxpool - fc1 - Relu - Dropout - fc2 - softmax
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = self.pool(self.relu(self.conv3(x)))
       
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = nn.functional.softmax(self.fc2(x), dim=1) 
        return x
3.1.4.2 test on small dataset¶
In [10]:
# test on small dataset to check if the model is able to learn
config["parameters"]["data_size"]["values"] = ['small']
config["parameters"]["epochs"]["values"] = [2000]
config["parameters"]["early_stop_epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-conv-layers_test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 98cqz57j
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/98cqz57j
wandb: Agent Starting Run: 614jgmpg with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_064547-614jgmpg
Syncing run desert-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/98cqz57j
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/98cqz57j
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/614jgmpg
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185930…

Run history:


Accuracy difference▅▁▁▅████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▅▁▁▅████████████████████████████████████
Training Loss█▇▅▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.46144

View run desert-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/614jgmpg
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_064547-614jgmpg/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.1.4.3 tune learning rate & batch size¶
  • train accuracy on small dataset reached 100% , indicated the re-defined model is able to learn, could be applied on normal dataset
  • find the optimal learning rate and batch size of this model architechture
    • learning rate: 0.1, 0.01, 0.001
    • batch size: 16, 32, 64, 128
    • use 100 epochs: as the model with first configuration (learning rate of 0.001, batch size of 128) had very large influctuation with 50 epochs, I increased the epochs to 100
In [12]:
# tune learning_rate & batch_size
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["epochs"]["values"] = [100]
config["parameters"]["learning_rate"]["values"] = [0.001,0.01,0.1]
config["parameters"]["batch_size"]["values"] = [128,64,32,16]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-lr-bs", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 7tmfsvmo
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
wandb: Agent Starting Run: sio95ncq with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_081942-sio95ncq
Syncing run earnest-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/sio95ncq
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.008 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.392585…

Run history:


Accuracy difference▆█▇▅▅▅▆▆▅▅▅▆▆▆▅▃▃▁▂▃▃▄▄▆▆▅▄▄▃▄▅▄▄▄▅▄▃▃▁▃
Test Accuracy▁▁▂▃▄▄▄▄▄▄▄▄▆▇███▇▇▆▅▄▄▃▃▂▂▃▃▃▄▄▅▅▅▅▅▅▆▇
Training Accuracy▁▁▂▃▄▄▄▄▄▄▄▄▆▇███▇▆▅▅▄▃▃▃▂▂▂▃▃▄▄▄▅▅▅▄▄▅▆
Training Loss██████████████████████▇▇▇▇▇▇▇▇▇▆▆▆▅▄▄▃▂▁

Run summary:


Accuracy difference-0.00668
Test Accuracy0.1598
Training Accuracy0.15312
Training Loss2.27367

View run earnest-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/sio95ncq
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_081942-sio95ncq/logs
wandb: Agent Starting Run: g2x17ssn with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_092502-g2x17ssn
Syncing run polar-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/g2x17ssn
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.008 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.392695…

Run history:


Accuracy difference▆▆▄▇█▄▄▃▂▃▃▄▃▃▃▄▁▂▃▄▅▂▄▁▂▂▃▂▄▄▃▄▅▄▃▃▅▅▅▃
Test Accuracy▁▂▂▂▂▃▃▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████
Training Accuracy▁▁▂▂▂▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇██████
Training Loss█████▇▆▆▅▅▅▅▄▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0316
Test Accuracy0.59
Training Accuracy0.5584
Training Loss1.90368

View run polar-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/g2x17ssn
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_092502-g2x17ssn/logs
wandb: Agent Starting Run: 0jm10odx with config:
wandb: 	batch_size: 128
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_103038-0jm10odx
Syncing run vivid-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/0jm10odx
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▁▄▅▃▆▅▃▄▅▄▅▄▄▇▅▅▅▇▇▆▇▅▆▇▆▆█▆▇▇▇▇▆▇▇▆▇█▇▇
Test Accuracy▁▂▃▄▄▅▅▆▆▆▆▇▇▆▇▇▇▇▇▇▇█▇▇██▇█████████████
Training Accuracy▁▃▄▄▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇███████████████
Training Loss█▆▆▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.02314
Test Accuracy0.8018
Training Accuracy0.82494
Training Loss1.63614

View run vivid-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/0jm10odx
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_103038-0jm10odx/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: ivczmyrh with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_113725-ivczmyrh
Syncing run gallant-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ivczmyrh
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.421035…

Run history:


Accuracy difference█▆▇███▇▆▅▅▆▇▇▆▅▄▄▄▄▄▅▄▃▄▃▃▃▃▃▃▂▃▂▂▁▁▂▂▂▁
Test Accuracy▁▂▁▁▁▁▁▂▂▃▃▃▃▃▃▄▄▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇██████
Training Accuracy▁▂▁▁▁▁▁▂▂▃▃▃▃▃▃▃▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇██████
Training Loss█████████████▇▇▇▆▆▅▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.03644
Test Accuracy0.3657
Training Accuracy0.32926
Training Loss2.12616

View run gallant-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ivczmyrh
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_113725-ivczmyrh/logs
wandb: Agent Starting Run: 10bevwe9 with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_130427-10bevwe9
Syncing run major-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/10bevwe9
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▄▇▄▄▃▄▃▄▃▂▁▂▆▃▅▅▃▃▄▃▃▅▆▅▄▅▃▅▃▃▄█▄▇▅▅▅▅▇▄
Test Accuracy▁▁▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇█▇██████
Training Accuracy▁▁▃▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇█████████
Training Loss██▇▆▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02638
Test Accuracy0.6933
Training Accuracy0.66692
Training Loss1.79498

View run major-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/10bevwe9
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_130427-10bevwe9/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: m0j232q9 with config:
wandb: 	batch_size: 64
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_143042-m0j232q9
Syncing run icy-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/m0j232q9
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▂▃▃▄▅▃▄▃▇▅▄▄▄▄█▆▅▄▅▅▅▅▅▅▄▅▅▅▄▄▅▇▅▄▅▅▅▄▅
Test Accuracy▁▃▄▄▅▅▆▆▇▅▆▇▇▇▇▆▇▇█▇▇█▇█████████▇███████
Training Accuracy▁▃▄▄▅▅▆▆▆▇▇▇▇▇▇▇▇███████████████████████
Training Loss█▆▅▅▄▄▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00526
Test Accuracy0.7878
Training Accuracy0.78254
Training Loss1.67788

View run icy-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/m0j232q9
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_143042-m0j232q9/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: ctnpm7y6 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_160508-ctnpm7y6
Syncing run breezy-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ctnpm7y6
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.016 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.781736…

Run history:


Accuracy difference▇▆▆▅▅▅▅▅▅▅█▇▄▄▄▃▃▂▂▂▂▂▁▁▁▂▂▁▂▁▂▂▂▂▂▂▂▂▂▂
Test Accuracy▁▂▃▃▃▃▃▃▃▄▃▂▄▄▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Accuracy▁▂▃▃▃▃▃▃▃▄▃▃▃▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇███████
Training Loss███████████▇▆▅▅▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.03142
Test Accuracy0.4189
Training Accuracy0.38748
Training Loss2.06981

View run breezy-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/ctnpm7y6
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_160508-ctnpm7y6/logs
wandb: Agent Starting Run: 5ucpygf4 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_175325-5ucpygf4
Syncing run stellar-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/5ucpygf4
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.420974…

Run history:


Accuracy difference▅▂▃▂▃▃▁▂▁▄▃▃▂▂▁▅▂▄▅▆▅▃▃▃▅▅▅▆▄▄▄▆▅▅▇▅█▇█▇
Test Accuracy▁▂▃▃▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Accuracy▁▁▃▃▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss██▇▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00224
Test Accuracy0.7632
Training Accuracy0.76096
Training Loss1.70115

View run stellar-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/5ucpygf4
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_175325-5ucpygf4/logs
wandb: Agent Starting Run: t70i2zz1 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_194431-t70i2zz1
Syncing run fragrant-sweep-9 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/t70i2zz1
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▄▁▃▄▂▃▃▄▃▇▅▂▄▄▃▄▄▄▃▄▄▅▁▅▅▅▄▆▅█▆▆▆▆▆
Test Accuracy▃▅▅▆▆▆▇▇▇▇▇█████████▇▇█▆▅▅▃▂▃▁▁▁▁▁▁
Training Accuracy▃▄▅▅▆▆▇▇▇▇████████████▇▇▆▅▂▂▃▂▁▁▁▁▁
Training Loss▆▅▄▃▃▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▂▂▃▄▇▇▆▇█████

Run summary:


Accuracy difference-0.0002
Test Accuracy0.1015
Training Accuracy0.1013
Training Loss2.35984

View run fragrant-sweep-9 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/t70i2zz1
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_194431-t70i2zz1/logs
wandb: Agent Starting Run: 83b57mw5 with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_202907-83b57mw5
Syncing run young-sweep-10 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/83b57mw5
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.185756…

Run history:


Accuracy difference█▇▇▇█▅▄▃▃▃▂▂▃▂▂▂▁▂▂▃▂▂▂▂▄▂▁▂▃▁▂▂▂▃▃▄▂▂▄▃
Test Accuracy▁▁▁▂▂▂▃▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇██████
Training Accuracy▁▁▁▂▂▂▃▄▄▄▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇██████
Training Loss█████▇▆▆▅▅▅▅▄▄▄▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02834
Test Accuracy0.5492
Training Accuracy0.52086
Training Loss1.9412

View run young-sweep-10 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/83b57mw5
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_202907-83b57mw5/logs
wandb: Agent Starting Run: 6f26m2r9 with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230429_225045-6f26m2r9
Syncing run glad-sweep-11 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/6f26m2r9
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.020 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.222901…

Run history:


Accuracy difference▄▁▂▁▂▁▁▃▅▅▃▂▃▃▄▄▅▆▄▆▆▆▅▇▆▆▆▆▆▇▇▇█▇██▇███
Test Accuracy▁▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█▇████████████████
Training Accuracy▁▂▃▄▄▄▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████████
Training Loss█▆▆▅▅▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01904
Test Accuracy0.8057
Training Accuracy0.82474
Training Loss1.63699

View run glad-sweep-11 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/6f26m2r9
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230429_225045-6f26m2r9/logs
wandb: Agent Starting Run: klpc0p70 with config:
wandb: 	batch_size: 16
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 100
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230430_011009-klpc0p70
Syncing run woven-sweep-12 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/klpc0p70
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▂▂▆█▆▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Test Accuracy▆▇█▆▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▄▇██▆▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Loss▄▂▁▁▃▇█████████████████

Run summary:


Accuracy difference0.0002
Test Accuracy0.1
Training Accuracy0.1002
Training Loss2.36095

View run woven-sweep-12 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-lr-bs/runs/klpc0p70
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230430_011009-klpc0p70/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.1.4.4 result of tuning (linked from wandb)¶
In [18]:
sweep = api.sweep(f"weiping-zhang/mc1-sgd-early-stop-lr-bs/sweeps/7tmfsvmo")
sweep.display(height=1024)
Out[18]:
True
3.1.4.5 interpretation¶
  • model with 3 convolutional layers + 2 fully connected layers, optimal learning rate is 0.01, optimal batch size is 32, because
    • learning rate 0.1 is too large, three of the four models failed to converge, large influctuation in test accuracy
    • learning rate 0.001 is too small, it is very slow to reach good performance as the other learning rates
    • learning_rate 0.01 is optimal in the three options
    • with the four batch_size options, the test accuracy increased with smaller batch size, but the training time is also larger.
    • with batch_size of 32, model has the best balance between computing time and test accuracy
    • therefore optimal batch size of this architecture is 32
3.1.4.6 use 500 epochs with optimal learning rate 0.01 & batch size 32¶
In [13]:
# update learning rate & batch size, train on normal dataset
config["parameters"]["learning_rate"]["values"] = [0.01]
config["parameters"]["batch_size"]["values"] = [32]
config["parameters"]["epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv_layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: c56mufxw
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/c56mufxw
wandb: Agent Starting Run: y047mnbg with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 500
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230430_014253-y047mnbg
Syncing run warm-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/c56mufxw
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/sweeps/c56mufxw
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/y047mnbg
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▂▁▁▁▃▂▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇█▇▇█▇▇███
Test Accuracy▁▃▄▅▆▆▇▇▇▇▇▇▇███████████████████████████
Training Accuracy▁▃▄▅▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████████████████
Training Loss█▆▅▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.08768
Test Accuracy0.841
Training Accuracy0.92868
Training Loss1.53325

View run warm-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/y047mnbg
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230430_014253-y047mnbg/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [19]:
run = api.run("weiping-zhang/mc1-sgd-early-stop-n-filters/y047mnbg")
run.display(height=1024)
Out[19]:
True

3.1.5 with 4 convolutional layers + 2 fc layers¶

3.1.5.1 re-define 2D CNN architecture¶
In [20]:
# use 4 convolutional layers
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)  
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3) 
        self.conv3 = nn.Conv2d(64,256,kernel_size=3) 
        self.conv4 = nn.Conv2d(256,512,kernel_size=3)  
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(512 * 2 * 2, 128) 
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - Maxpool - CONV3 - Relu - Maxpool - CONV3 - Relu - Maxpool - fc1 - Relu - Dropout - fc2 - softmax
        
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = self.pool(self.relu(self.conv3(x)))
        x = self.pool(self.relu(self.conv4(x)))
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = nn.functional.softmax(self.fc2(x), dim=1) 
        return x
3.1.5.2 test on small dataset¶
In [14]:
# test on small dataset to check if the model is able to learn
config["parameters"]["data_size"]["values"] = ['small']
config["parameters"]["epochs"]["values"] = [2000]
config["parameters"]["early_stop_epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-conv-layers_test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: ngg9eqhr
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/ngg9eqhr
wandb: Agent Starting Run: tl0dsgj4 with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 2000
wandb: 	epochs: 5000
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230430_112821-tl0dsgj4
Syncing run jumping-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/ngg9eqhr
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/ngg9eqhr
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/runs/tl0dsgj4
wandb: Ctrl + C detected. Stopping sweep.
In [21]:
sweep = api.sweep(f"weiping-zhang/mc1-sgd-early-stop-n-conv-layers_test/sweeps/ngg9eqhr")
sweep.display(height=1024)
Out[21]:
True
3.1.5.3 tune learning rate & batch size¶
  • train accuracy on small dataset reached 100% , indicated the re-defined model is able to learn, could be applied on normal dataset
  • find the optimal learning rate and batch size of this model architechture
    • learning rate: 0.1, 0.01, 0.001
    • batch size: 16, 32, 64, 128
    • use 50 epochs: as section 3.1.1.2 showed that training with 50 epochs is already helpful to select optimal learning rate and batch size
In [ ]:
# tune learning_rate & batch_size
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["epochs"]["values"] = [50]
config["parameters"]["learning_rate"]["values"] = [0.001,0.01,0.1]
config["parameters"]["batch_size"]["values"] = [128,64,32,16]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-lr-bs", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
3.1.5.4 result of tuning (linked from wandb)¶
In [8]:
%wandb weiping-zhang/mc1-sgd-early-stop-lr-bs/reports/4-conv-layer-tune-lr-bs--Vmlldzo0NTI0NjQz -h 1024
In [9]:
%wandb weiping-zhang/mc1-sgd-early-stop-lr-bs/reports/4-conv-layer-lr-0-01-bs--Vmlldzo0NDg4Mjg5 -h 1024
3.1.5.5 interpretation¶
  • model with 4 convolutional layers + 2 fully connected layers, optimal learning rate is 0.01, optimal batch size is 32, because
    • learning rate 0.1 is too large, three of the four models failed to converge, large influctuation in test accuracy. Though model with learning rate 0.1 & batch size 128 seems have good performance, however here is only 50 epochs, it may fail to converge later as other three models with same learning rate.
    • learning rate 0.001 is too small, it is very slow to reach good performance comparing to models with the smaller learning rates
    • learning_rate 0.01 is optimal in these three options
    • with the four batch_size options, the test accuracy increased with smaller batch size, but the training time is also larger.
    • with batch_size of 16, model has much better performance than others, but the computing time is very long, where with batch size of 32, model has a better balance between computing time and test accuracy
    • I will train models using 500 epochs with the two batch size to observe which one is better.
3.1.5.6 use 500 epochs with optimal learning rate 0.01 & batch size 16 or 32¶

from the report below, we could see that,

  • after 150 epochs, the test accuracy in two models are very similar
  • the accuracy difference with batch size 16 is much larger than with batch size 32, though after 400 epochs it is opposite (but very likely we will not need so many epochs).
  • computing time with batch size 32 is around 50% of with batch size 16 So I will select batch size 32 for the 4-conv-layer model
In [ ]:
# update learning rate & batch size, train on normal dataset
config["parameters"]["learning_rate"]["values"] = [0.01]
config["parameters"]["batch_size"]["values"] = [16,32]
config["parameters"]["epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv_layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
In [10]:
%wandb weiping-zhang/mc1-sgd-early-stop-n_conv_layers/reports/optimal-4-conv-layer-models--Vmlldzo0NTI0NzAz -h 1024

3.1.6 summary: optimal number of layers¶

  • model with more convolutional layers have also higher accuracy in both train and test set.
  • test accuracy: with 4 or 3 conv-layers >> with 2 or 1 conv-layers >> with 0 conv-layer
  • accuracy difference: with 4 conv-layers >> with 3 conv-layers >> with 0 or 1 or 2 conv-layers
    • both models with 3 and 4 conv-layers are moderate overfitting. But this is quite common in deep learning. It could be later improved by applying regularization and stop model early at a suitable epoch.
  • computing cost: 4 conv-layers > 2 or 3 conv-layers >> 0 or 1 conv-layers

My final choice is between with 3 or 4 conv-layer, as their performance is much better than other three.

  • 4-conv-layer-model has only slightly higher test accuracy than with 3 conv-layers, but much worse overfitting (larger than 0.1) than with 3-conv-layer-model (0.08768). with 4 conv layers, the computational time is as well larger than with 3 conv layers. In this case, I prefer a simplier model.
  • So my choice is using 3 convolutional layers, due to its high performance on both train and testset, and acceptable computational time (similar to with 2 conv layers), model is moderate overfitting, but I will improve this later by applying suitable regularization algorithms.
In [11]:
%wandb weiping-zhang/mc1-sgd-early-stop-n_conv_layers/reports/how-many-convolutional-layers-is-optimal---Vmlldzo0NDg3NjI1 -h 1024

3.2 How many fully connected layers for the 3-conv-layer model?¶

In the last steps, the optimal model was using 2 fully connected layers, so i will further test with

  • 1 fully connected layer
  • 3 fully connected layers

3.2.1 with 1 fully connected layer¶

3.2.1.1 re-define model architecture¶
In [26]:
# use 3 convolutional layers + 1 fully connected layer
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(3, 32, kernel_size=3) 
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3) 
        self.conv3 = nn.Conv2d(64,256,kernel_size=3) 
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(256 * 6 * 6, 10) 
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - Maxpool - CONV3 - Relu - Maxpool - fc1 - softmax
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = self.pool(self.relu(self.conv3(x)))
        x = torch.flatten(x, start_dim=1)
        x = nn.functional.softmax(self.fc1(x), dim=1) 
        return x
3.2.1.2 result¶
In [43]:
config = {
    "method": "grid",
    "metric": {'name': 'Test Accuracy', 'goal': 'maximize'},
    "parameters": {
        "data_size":{"values":["normal"]},
        "early_stop_epochs":{"values":[20]},
        "out_1":{"values":[64]},
        "out_2":{"values":[128]},
        "out_3":{"values":[512]},
        "optimizer": {"values": ["sgd"]},
        "learning_rate": {"values": [0.01]},
        "batch_size": {"values": [32]},
        "momentum": {"values": [0]},    
        "epochs": {"values": [80]},    
        "dropout_rate": {"values": [0]},     # dropout = 0 -> drop out no neurons 
        "regularization": {"values": ["none"]},  # no regularization
        "regularization_strength": {"values": [0]}}}
#
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n_conv-layers", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 3yzbdzoc
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers/sweeps/3yzbdzoc
wandb: Agent Starting Run: 4nf3wmdk with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	regularization: none
wandb: 	regularization_strength: 0
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230509_222209-4nf3wmdk
Syncing run sparkling-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers/sweeps/3yzbdzoc
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers/sweeps/3yzbdzoc
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers/runs/4nf3wmdk
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▁▁▃▅▆▃▂▃▁▃▂▅▃▃▁▂▆▄▂▅▃▃▃▅▅▅▃▄▇█▄▅▆▄▆▅▅▅▇▅
Test Accuracy▁▃▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Accuracy▁▃▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0259
Test Accuracy0.7485
Training Accuracy0.7226
Training Loss1.74148

View run sparkling-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-conv-layers/runs/4nf3wmdk
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230509_222209-4nf3wmdk/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [12]:
run = api.run("/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/4nf3wmdk")
print(run.display(height=1024))
/opt/anaconda3/lib/python3.9/site-packages/IPython/core/display.py:419: UserWarning: Consider using IPython.display.IFrame instead
  warnings.warn("Consider using IPython.display.IFrame instead")
True

3.2.2 with 3 fully connected layers¶

3.2.2.1 re-define model architecture¶
In [28]:
# use 3 convolutional layers with 3 fully connected layers
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(3, 32, kernel_size=3) 
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3) 
        self.conv3 = nn.Conv2d(64,256,kernel_size=3) 
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        
        self.fc1 = nn.Linear(256 * 6 * 6, 128) 
        self.fc2 = nn.Linear(128, 32)
        self.fc3 = nn.Linear(32, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - Maxpool - CONV3 - Relu - Maxpool - fc1 - Relu - Dropout
        # - fc2 - Relu - Dropout - fc3 - softmax
        
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = self.pool(self.relu(self.conv3(x)))
       
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.dropout(self.relu(self.fc2(x)))
                             
        x = nn.functional.softmax(self.fc3(x), dim=1) 
        return x
3.2.2.2 result¶
In [13]:
run = api.run("/weiping-zhang/mc1-sgd-early-stop-n_conv_layers/runs/xqb9ydp4")
run.display(height=1024)
Out[13]:
True

3.2.3 summary - (recap with 2 fully connected layers)¶

In [14]:
%wandb weiping-zhang/mc1-sgd-early-stop-n_conv_layers/reports/how-many-fully-connected-layers--Vmlldzo0NDg4NjE0 -h 1024

3.2.4 interpretation & optimal number of fc layers¶

  • compare the three 3-conv models with 1 fc layer, 2 fc layers, 3 fc layers (80 epochs):

    • train accuracies: 0.7226, 0.7285, 0.6993
    • test accuracies: 0.7485, 0.7318, 0.6997
    • the performance with 2 fc layers and 1 fc layer is better than with 3 fc layers
    • but the accuracy line with 1 fc layer is much flatter than with 2 fc layers, this means when use more epochs, the model with 2 fc layers will likely reach much better accuracy.

      --> use the architecture 3 convolutional layers + 2 fc layers

3.3 How many filters in each convolutional layer?¶

  • When using more filters in convolutional layers,

    • +++: the model becomes more capable to capture features and patterns. The model could learn more diverse and detailed representations, this could potentially improve its ability to distinguish between different classes or categories
    • ---: it will increase model complexity and the overfitting risk. The computational cost will increase too.
  • It's important to balance the number of filters with the size of the dataset and the overall model architecture.

  • in the last steps, I have used 32 - 64 - 256 filters in the three convolutional layers

  • here I will test
    • doubled number of filters 64 - 128 - 512, and
    • half number of filters 16 - 32 - 128

3.3.1 reform the network structure using more configurations¶

In [32]:
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, config.out_1, config.kernel_size, stride = config.stride, padding = config.padding) 
        self.conv2 = nn.Conv2d(config.out_1, config.out_2, config.kernel_size, stride = config.stride, padding = config.padding) 
        self.conv3 = nn.Conv2d(config.out_2,config.out_3,config.kernel_size, stride = config.stride, padding = config.padding) 
        self.relu = nn.ReLU()
        if config.pool_type == 'max':
            self.pool = nn.MaxPool2d(2)
        elif config.pool_type == 'avg':
            self.pool = nn.AvgPool2d(2)
        # output size of each layer (including pooling) 
        self.size1 = math.floor((32 + 2 * config.padding - config.kernel_size) / config.stride + 1) 
        self.size2 = math.floor(math.floor((self.size1 + 2 * config.padding - config.kernel_size) / config.stride + 1)/2)
        self.size3 = math.floor(math.floor((self.size2 + 2 * config.padding - config.kernel_size) / config.stride + 1)/2)
    
        self.fc1 = nn.Linear(config.out_3*(self.size3)**2, 128)  
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])
    def forward(self, x):
        # CONV1 - Relu - CONV2 - Relu - pool - CONV3 - Relu - pool - fc1 - Relu - Dropout - fc2 - softmax
        
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = self.pool(self.relu(self.conv3(x)))
       
        x = torch.flatten(x, start_dim=1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        # apply the softmax activation function to the output of the second fully connected layer
        x = nn.functional.softmax(self.fc2(x), dim=1) 
        return x
3.3.1.1 test the reformed network on small dataset¶
In [10]:
config = {
    "method": "grid",
    "metric": {'name': 'Test Accuracy', 'goal': 'maximize'},
    "parameters": {
        "data_size":{"values":["small"]},
        "early_stop_epochs":{"values":[200]},
        "out_1":{"values":[64]},
        "out_2":{"values":[128]},
        "out_3":{"values":[512]},
        "kernel_size":{"values":[3]},
        "stride":{"values":[1]},
        "padding":{"values":[0]},
        "pool_type":{"values":["max"]},
        "optimizer": {"values": ["sgd"]},
        "learning_rate": {"values": [0.01]},
        "batch_size": {"values": [32]},
        "momentum": {"values": [0]},    
        "epochs": {"values": [2000]},    
        "dropout_rate": {"values": [0]},     # dropout = 0 -> drop out no neurons 
        "regularization": {"values": ["none"]},  # no regularization
        "regularization_strength": {"values": [0]}}}

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-filters-test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 6lza4xpb
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test/sweeps/6lza4xpb
wandb: Agent Starting Run: qw87t26p with config:
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 200
wandb: 	epochs: 2000
wandb: 	input_fc1: 18432
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230501_002713-qw87t26p
Syncing run vivid-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test/sweeps/6lza4xpb
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test/sweeps/6lza4xpb
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test/runs/qw87t26p
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▁▁▁▁▁▁▁▅▅██████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▁▁▁▅▅██████████████████████████████
Training Loss██▆▅▅▅▄▄▃▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss1.4619

View run vivid-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters-test/runs/qw87t26p
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230501_002713-qw87t26p/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.

- the train accuracy reached 100%, this indicated the reformed network and configuration works, model could learn the patterns from data ¶


3.3.2 with 64 - 128 - 512 filters¶

In [8]:
# by normal dataset
config["parameters"]["data_size"]["values"] = ['normal']
config["parameters"]["epochs"]["values"] = [300]
config["parameters"]["early_stop_epochs"]["values"] = [20]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-filters", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: ie8vacvy
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/ie8vacvy
wandb: Agent Starting Run: 911sgix1 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 300
wandb: 	input_fc1: 18432
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230430_122049-911sgix1
Syncing run ruby-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/ie8vacvy
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/ie8vacvy
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/runs/911sgix1
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.005 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.793742…

Run history:


Accuracy difference▁▂▁▃▂▂▂▃▃▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇█▇▇███
Test Accuracy▁▂▃▄▅▆▆▆▇▇▇▇▇▇▇▇████████████████████████
Training Accuracy▁▂▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████████
Training Loss█▇▆▅▅▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.07754
Test Accuracy0.8501
Training Accuracy0.92764
Training Loss1.53503

View run ruby-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/runs/911sgix1
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230430_122049-911sgix1/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.

3.3.3 with 16 - 32 - 128 filters¶

In [9]:
# use half number of filters for each convolutional layer: 16 - 32 - 128 
config["parameters"]["out_1"]["values"] = [16]
config["parameters"]["out_2"]["values"] = [32]
config["parameters"]["out_3"]["values"] = [128]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-n-filters", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 3ed6xi0q
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/3ed6xi0q
wandb: Agent Starting Run: 1wja1hs8 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 300
wandb: 	input_fc1: 4608
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 16
wandb: 	out_2: 32
wandb: 	out_3: 128
wandb: 	padding: 0
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
wandb: WARNING WANDB_NOTEBOOK_NAME should be a path to a notebook file, couldn't find /Users/rheinpark/Desktop/deep learning.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230430_194540-1wja1hs8
Syncing run lilac-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/3ed6xi0q
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/sweeps/3ed6xi0q
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/runs/1wja1hs8
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.421331…

Run history:


Accuracy difference▁▁▂▄▃▂▂▂▃▃▄▃▄▃▅▅▅▅▅▆▆▆▆▆▆▆▆▇▆▇▆█▇▇█▇▇██▇
Test Accuracy▁▃▄▄▅▆▆▆▆▇▇▇▇▇▇▇▇▇█▇████████████████████
Training Accuracy▁▂▄▄▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Loss█▇▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.0348
Test Accuracy0.8149
Training Accuracy0.8497
Training Loss1.61161

View run lilac-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-n-filters/runs/1wja1hs8
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230430_194540-1wja1hs8/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.

3.3.4 plot all three runs (recap 32 - 64 - 256 filters)¶

In [15]:
%wandb weiping-zhang/mc1-sgd-early-stop-n-filters/reports/How-many-filters-in-convolutional-layers---Vmlldzo0NTI1MTAx -h 1024

3.3.5 interpretation & optimal number of filters (64 - 128 - 512 filters)¶

  • I will use 64 - 128 - 512 filters in convolutional layers for the further steps, because
    • model has the highest accuracy on both train and test set in all three models
    • though the model is also slightly stronger overfitting (indicated by accuracy difference) than other two, this could be interesting and improved by introducing the regularization in the later steps
    • the computing time is larger than with less filters, but not that extremely large
  • This model has the best balance in performance, overfitting risk, computational cost

3.4 tune convolutional hyperparameter¶

3.4.1 tune kernel size, stride, padding¶

  • I will tune the three hyperparameter together, because they are all interrelated.
  • kernel size: dimensions of the filter applied in each convolutional layer, I will test 2, 3, 4 (use same kernel size for all three convolutional layers)

    • with larger kernel size:
      • convolutional layer could capture more information from input. This could be especially beneficial for the tasks require capturing larger patterns. However, this may result in the loss of fine details or local patterns.
      • increased the computational complexity
      • if kernel size is too large, model may have bad generalization
    • with smaller kernel size:
      • convolutional layer will capture fine details, but this might also make the model more sensitive to noise and more computationally expensive due to increased number of operations.
    • Therefore, an optimal kernel size should have the good balance between detail and computational efficiency.
  • stride: step size that the filter moves for each convolution, I will test 1, 2 (use same stride for all three convolutional layers)

    • smaller strides may capture more details but be more computationally expensive, and may lead overfitting
    • larger strides will cover a smaller portion of input, this may potentially cause missing important patterns, therefore result in reduced accuracy and reduced computational time.
  • padding: add extra values (usually zero) around the input tensor to ensure that the spatial information at the edges of the input is also considered during the convolution operation. I will test 0, 1, 2 (use same padding for all three convolutional layers).

    • however, excessive padding might introduce too many non-informative pixels and might cause model learn less useful features.
    • helps model to preserve spatial information in the output feature
    • adds additional pixels around the input, so that the convolutional layers could capture information from the edges and corners of the input data. However, this may also introduce noise or irrelevant information from borders.
    • increases computational cost, as the added pixels require extra computations
  • Not all the configuration combinations will work, e.g. padding = 0, stride = 2, kernel_size = 3, image size will somehow not large enough for further processing, so the training will stop.

In [10]:
config = {
    "method": "grid",
    "metric": {'name': 'Test Accuracy', 'goal': 'maximize'},
    "parameters": {
        "data_size":{"values":["normal"]},
        "early_stop_epochs":{"values":[20]},
        "out_1":{"values":[64]},
        "out_2":{"values":[128]},
        "out_3":{"values":[512]},
        "kernel_size":{"values":[2,3,4]},
        "stride":{"values":[1,2]},
        "padding":{"values":[0,1,2]},
        "pool_type":{"values":["max"]},
        "optimizer": {"values": ["sgd"]},
        "learning_rate": {"values": [0.01]},
        "batch_size": {"values": [32]},
        "momentum": {"values": [0]},    
        "epochs": {"values": [80]},    
        "dropout_rate": {"values": [0]},     # dropout = 0 -> drop out no neurons 
        "regularization": {"values": ["none"]},  # no regularization
        "regularization_strength": {"values": [0]}}}

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-kernel-stride-pad", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: p1kesjyt
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
wandb: Agent Starting Run: ea148kth with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230501_193908-ea148kth
Syncing run skilled-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ea148kth
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183307…

Run history:


Accuracy difference█▃▃▂▂▃▁▃▃▄▅▃▅▄▄▂▄▄▅▄▆▅▄▇▅▆▅▃▅▆▇▆▆▆▆▆█▅▆▇
Test Accuracy▁▃▄▄▄▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇█████████████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▆▅▅▅▄▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0066
Test Accuracy0.7287
Training Accuracy0.7221
Training Loss1.74131

View run skilled-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ea148kth
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230501_193908-ea148kth/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: w0ona7dv with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230501_212956-w0ona7dv
Syncing run mild-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/w0ona7dv
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.420701…

Run history:


Accuracy difference▄▃▁▆▃▃▃▃▂▂▃▃▃▄▄▄▄▄▅▄▄▅▅▅▆▆▆▆▆▇▆▇▆▇█▆▇▇▇▇
Test Accuracy▁▁▃▂▃▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█▇▇█████
Training Accuracy▁▁▂▂▂▃▃▃▄▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇███████
Training Loss████▇▇▆▆▆▆▅▅▅▄▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference0.0312
Test Accuracy0.4689
Training Accuracy0.5001
Training Loss1.9591

View run mild-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/w0ona7dv
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230501_212956-w0ona7dv/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: ouzv6wub with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230501_224611-ouzv6wub
Syncing run hearty-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ouzv6wub
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.230868…

Run history:


Accuracy difference▂▁▄▄▃▄▄▄▂▁▃▃▂▂▃▃▃▄▂▂▅▆▂▂▃█▃▃▅▃▅▄▅▃▅▅▅▅▅▅
Test Accuracy▁▃▄▄▄▄▄▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█▇█████████
Training Accuracy▁▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▆▆▅▅▅▅▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00362
Test Accuracy0.6462
Training Accuracy0.64258
Training Loss1.81947

View run hearty-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ouzv6wub
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230501_224611-ouzv6wub/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: dpen14jd with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_005713-dpen14jd
Syncing run super-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/dpen14jd
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.242222…

Run history:


Accuracy difference▆▄█▆▅▅▄▄▄▃▃▃▃▂▂▁▂▂▂▁▄▁▁▃▄▂▃▂▁▁▃▂▂▂▂▂▃▄▃▂
Test Accuracy▁▁▁▂▂▃▃▄▄▄▄▄▅▅▅▅▆▆▆▆▆▆▇▆▆▇▇▇▇▇▇▇▇▇██████
Training Accuracy▁▁▂▂▂▃▃▃▄▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████
Training Loss████▇▇▆▆▆▆▅▅▅▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02684
Test Accuracy0.4745
Training Accuracy0.44766
Training Loss2.01163

View run super-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/dpen14jd
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_005713-dpen14jd/logs
wandb: Agent Starting Run: y5oyt0af with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_022218-y5oyt0af
Syncing run soft-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/y5oyt0af
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183274…

Run history:


Accuracy difference▁▂▅▃▄▂▅▄▃▂▄▂▂▄▂▄▅▃▃▃▃▃▆▂▆▅▄▄▄▅▄▅█▅▆▄▆▆▆▅
Test Accuracy▁▃▃▄▄▅▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇██▇███████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▅▅▅▅▄▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00798
Test Accuracy0.7335
Training Accuracy0.72552
Training Loss1.73804

View run soft-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/y5oyt0af
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_022218-y5oyt0af/logs
wandb: Agent Starting Run: ubhpa9ps with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 2
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_053402-ubhpa9ps
Syncing run peach-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ubhpa9ps
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183299…

Run history:


Accuracy difference█▇▅▅▅▂▆▄▃▄▄▃▂▃▃▆▆▃▃▄▅▂▁▃▃▃▃▃▂▃▁▁▄▁▂▇▃▅▂▃
Test Accuracy▁▁▂▂▂▂▃▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██▇████
Training Accuracy▁▁▁▂▂▂▃▃▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████
Training Loss█████▇▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁

Run summary:


Accuracy difference-0.03082
Test Accuracy0.5668
Training Accuracy0.53598
Training Loss1.92557

View run peach-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ubhpa9ps
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_053402-ubhpa9ps/logs
wandb: Agent Starting Run: vgvtu8ts with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_072706-vgvtu8ts
Syncing run atomic-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/vgvtu8ts
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▂▁▅▂▂▅▄▂▃▂▂▂▃▄▅▄▅▄▃▄▃▇▄▄▅▅▆▇▅▆▆▆▆▆▇▆▇██▇
Test Accuracy▁▃▃▄▄▄▄▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Accuracy▁▂▃▄▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▆▅▅▅▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00436
Test Accuracy0.7793
Training Accuracy0.77494
Training Loss1.68891

View run atomic-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/vgvtu8ts
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_072706-vgvtu8ts/logs
wandb: Agent Starting Run: 5vv4wr47 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_093659-5vv4wr47
Syncing run volcanic-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/5vv4wr47
/opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…
View run volcanic-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/5vv4wr47
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_093659-5vv4wr47/logs
Run 5vv4wr47 errored: RuntimeError('Given input size: (512x1x1). Calculated output size: (512x0x0). Output size is too small')
wandb: ERROR Run 5vv4wr47 errored: RuntimeError('Given input size: (512x1x1). Calculated output size: (512x0x0). Output size is too small')
wandb: Agent Starting Run: o3usswyl with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_093720-o3usswyl
Syncing run upbeat-sweep-9 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/o3usswyl
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183265…

Run history:


Accuracy difference▁▁▅▄▄▃▃▃▄▂▁▂▆▃▂▂▄▆▂▅▄▄▄▇▅▇▇▅▇█▆▆▇▇▇▇▇█▇█
Test Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▇▇▆▇▇▇▇▇▇▇▇▇█▇▇██████████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▆▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00434
Test Accuracy0.7768
Training Accuracy0.78114
Training Loss1.68231

View run upbeat-sweep-9 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/o3usswyl
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_093720-o3usswyl/logs
wandb: Agent Starting Run: lgq1fa05 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_123340-lgq1fa05
Syncing run soft-sweep-10 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/lgq1fa05
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183311…

Run history:


Accuracy difference█▆▆▃▅▃▄▃▁▂▅▂▂▂▁▂▇▂▃▂▃▆▂▄▅▃▇▅▇█▅▇▅▅▄▆▅▅▅▅
Test Accuracy▁▁▂▂▂▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇███████
Training Accuracy▁▁▂▁▂▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇███████
Training Loss████▇▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01806
Test Accuracy0.6388
Training Accuracy0.62074
Training Loss1.84061

View run soft-sweep-10 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/lgq1fa05
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_123340-lgq1fa05/logs
wandb: Agent Starting Run: xmfxs0mk with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_140530-xmfxs0mk
Syncing run morning-sweep-11 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/xmfxs0mk
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183311…

Run history:


Accuracy difference▆▁▄▃▄▃▂▄▂▅▁▂▂▂▁▃▂▃▆▄▅▃▅▄▄▄▆▅▅▄▇▆▇▇▇▆█▇▇▇
Test Accuracy▁▃▄▄▄▅▅▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█▇█████████████
Training Accuracy▁▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00726
Test Accuracy0.7715
Training Accuracy0.77876
Training Loss1.6852

View run morning-sweep-11 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/xmfxs0mk
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_140530-xmfxs0mk/logs
wandb: Agent Starting Run: 87i7ffr5 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 3
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_170419-87i7ffr5
Syncing run apricot-sweep-12 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/87i7ffr5
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.005 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.793764…

Run history:


Accuracy difference▅▁█▂▃▂▂▂▂▂▁▂▃▃▂▁▄▂▂▃▂▄▂▃▅▂▃▄▄▂▃▃▃▄▃▅▂▃▄▃
Test Accuracy▁▂▁▃▃▄▄▄▅▅▅▆▆▆▆▆▆▆▆▆▇▆▇▇▇▇▇▇▇▇▇▇████████
Training Accuracy▁▁▂▂▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss███▇▆▆▆▅▅▅▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.02188
Test Accuracy0.6112
Training Accuracy0.58932
Training Loss1.87198

View run apricot-sweep-12 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/87i7ffr5
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_170419-87i7ffr5/logs
wandb: Agent Starting Run: qpcy0dqh with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_182710-qpcy0dqh
Syncing run classic-sweep-13 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/qpcy0dqh
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▇▁▃▃▅▃▄▃▂▅▃▅▅▃▅▅▄▅▅▄▅▅▅▆▄▅▆▅▆▇█▆▇▆▇▇▅▆▆▇
Test Accuracy▁▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇████
Training Accuracy▁▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇████
Training Loss█▇▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁

Run summary:


Accuracy difference0.00012
Test Accuracy0.7597
Training Accuracy0.75982
Training Loss1.70484

View run classic-sweep-13 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/qpcy0dqh
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_182710-qpcy0dqh/logs
wandb: Agent Starting Run: eunwenij with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 0
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_202704-eunwenij
Syncing run fanciful-sweep-14 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/eunwenij
Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.419562…
View run fanciful-sweep-14 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/eunwenij
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_202704-eunwenij/logs
Run eunwenij errored: RuntimeError("Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size")
wandb: ERROR Run eunwenij errored: RuntimeError("Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size")
wandb: Agent Starting Run: dhlng4c8 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_202725-dhlng4c8
Syncing run hopeful-sweep-15 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/dhlng4c8
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.230971…

Run history:


Accuracy difference▂▁▂▃▂▂▂▃▂▂▃▃▃▃▅▄▃▆▃▃▃▄▄▅▅▄▅▅▆▅▅▅▆▇▅▅▆▆█▆
Test Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▇▆▇▇▇▇▇▇▇▇▇█▇███████████
Training Accuracy▁▃▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▆▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01162
Test Accuracy0.7835
Training Accuracy0.79512
Training Loss1.66855

View run hopeful-sweep-15 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/dhlng4c8
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_202725-dhlng4c8/logs
wandb: Agent Starting Run: 3q2yayxz with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230502_225305-3q2yayxz
Syncing run true-sweep-16 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/3q2yayxz
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.230914…

Run history:


Accuracy difference▂▃█▁▂▃▂▃▃▂▂▁▃▂▄▆▂▁▂▄▂▃▃▅▃▅▅▄▅▃▄▃▆▆▄▆▅▆▄▅
Test Accuracy▁▁▁▃▃▄▄▄▅▅▅▆▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇███▇███████
Training Accuracy▁▁▂▂▃▄▄▄▄▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss███▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00054
Test Accuracy0.6314
Training Accuracy0.63194
Training Loss1.82995

View run true-sweep-16 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/3q2yayxz
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230502_225305-3q2yayxz/logs
wandb: Agent Starting Run: gokbcy2m with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_001454-gokbcy2m
Syncing run winter-sweep-17 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/gokbcy2m
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▂▁▃▃▅▄▄▃▄▄▃▄▄▅▄▃▅▄▄▄▄▅▅▅▆▅▅▆▆▇▆▇▇▇█▇█▇▇▇
Test Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇█▇▇▇██████████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01202
Test Accuracy0.7816
Training Accuracy0.79362
Training Loss1.66955

View run winter-sweep-17 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/gokbcy2m
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_001454-gokbcy2m/logs
wandb: Agent Starting Run: 65ndosjg with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 2
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 2
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.0166693182332286, max=1.0))…
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_030202-65ndosjg
Syncing run colorful-sweep-18 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/65ndosjg
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.021 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▇▇▅▅▃▂▂▃▆▃▃▁▂▄▃▄▃▃▄▅▃▅▅▇▅▅▅▄▅▆▅▅▅▅▆▆▆▇██
Test Accuracy▁▁▂▃▃▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇█████████████
Training Accuracy▁▁▂▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss███▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00354
Test Accuracy0.6312
Training Accuracy0.63474
Training Loss1.82746

View run colorful-sweep-18 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/65ndosjg
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_030202-65ndosjg/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [16]:
sweep = api.sweep("weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/p1kesjyt")
sweep.display(height=1024)
Out[16]:
True

3.4.2 interpretation & optimal kernel size, stride, padding¶

  • optimal kernel size = 4, stride = 1, padding = 1, because

    • accuracy of all models with stride = 1 are significantly better than with stride = 2, though computatio time is also larger
    • combination of kernel size and padding:
      • accuracy of kernel size = 4 & padding = 1 or 2 is better than other models
      • with padding = 1, computational time is much shorter than with padding = 2
  • with optimal kernel size, stride, padding, i would test the difference between average pooling and maximum pooling

3.4.3 pooling types: average vs max¶

  • pooling is a down-sampling operation. It is usually applied after convolutional layers to reduce dimensionality and allow model to learn more robust features.
  • larger pooling size, such as 3x3 or 4x4, can reduce the spatial dimensions more aggressively. This can be beneficial if the input images are large or when the spatial details are less important for the task.
  • smaller pooling size, such as 2x2, is commonly used and often works well in many image classification tasks. It provides a good balance between downsampling and preserving spatial details.
  • each image in CIFAR10 dataset has 32 x 32 pixels (relative small), 2x2 pooling size would be suitable. So here I will skip tuning the pooling size, directly test different pooling types.
    • max pooling will extract the most prominent features like edges
    • average pooling will extract features uniformly, it may miss out certain specific features
3.4.3.1 average pooling¶
In [13]:
# select kernel_size = 4, stride = 1, padding = 1
# test avgerage pooling 
config['parameters']['kernel_size']['values']=[4] #411
config['parameters']['stride']['values']=[1] 
config['parameters']['padding']['values']=[1] 
config['parameters']['pool_type']['values']=['avg'] 
config['parameters']['regularization']['values']=['none'] 
config['parameters']['regularization_strength']['values']=[0] 
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-kernel-stride-pad", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: fve538oi
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/fve538oi
wandb: Agent Starting Run: ckbwpf9e with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: avg
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_062828-ckbwpf9e
Syncing run tough-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/fve538oi
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/sweeps/fve538oi
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ckbwpf9e
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183320…

Run history:


Accuracy difference▁▃▃▃▅▃▅▄▃▄▅▅▅▅▅▆▅▆▅▅▅▆▆▆▅█▆▆▇▇▇▆▇▇▇█▇▇█▇
Test Accuracy▁▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇█▇███████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇███████
Training Loss█▆▆▅▅▅▅▅▄▄▄▄▄▄▄▃▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00366
Test Accuracy0.7016
Training Accuracy0.69794
Training Loss1.76498

View run tough-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/runs/ckbwpf9e
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_062828-ckbwpf9e/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.4.3.2 compare and interpret avg vs max (recap)¶
  • model performance with max pooling is much better than average pooling
  • computational time are similar
  • max pooling is more suitable for our dataset.
In [17]:
%wandb weiping-zhang/mc1-sgd-early-stop-kernel-stride-pad/reports/avg-vs-max-pooling--Vmlldzo0NDg4ODY5 -h 1024

3.4.4 use 500 epochs with optimal kernel size & stride & padding & pooling¶

  • the model reaches 90% accuracy (train) after 170 epochs, where the accuracy of test set is 82.8%
  • the large accuracy difference indicated the model is overfitting.
  • So I will apply l1, l2 regularization and dropout to mitigate the overfitting in the next steps
In [28]:
# train with 500 epochs to check how good will the accuray arrive
config['parameters']['pool_type']['values']=['max'] 
config["parameters"]["epochs"]["values"] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 94cghr2x
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/94cghr2x
wandb: Agent Starting Run: tyv3xsy5 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 500
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_203106-tyv3xsy5
Syncing run scarlet-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/94cghr2x
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/94cghr2x
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/tyv3xsy5
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.230971…

Run history:


Accuracy difference▁▁▂▂▃▃▄▄▄▄▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇████████
Test Accuracy▁▃▅▆▆▇▇▇▇▇▇▇████████████████████████████
Training Accuracy▁▃▄▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇██████████████████████
Training Loss█▆▅▄▄▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.11746
Test Accuracy0.858
Training Accuracy0.97546
Training Loss1.48637

View run scarlet-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/tyv3xsy5
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_203106-tyv3xsy5/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [18]:
sweep = api.sweep("weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/94cghr2x")
sweep.display(height=1024)
Out[18]:
True

3.5 regularization¶

  • l1, l2 and dropout regularization are mostly frequently used regularization techniques in deep learning to improve model generalization
  • l1 regularization, also called lasso regularization.

    • it pushes less important weights towards zero, so it will have sparse weight matrix
    • l1 regularization is helpful for feature selection and simplification
    • the regularization term in the loss function is the sum of the absolute values of the weights multiplied by a regularization strength hyperparameter (lambda or alpha)
  • l2 regularization, also called ridge regularization

    • it encourages small weights for all features, discouraging large variations and reducing the impact of individual features on output
    • l2 prevents large weights to avoid overly complex models
    • regularization term in the loss function is the sum of the squared values of the weights multiplied by a regularization strength hyperparameter (lambda or alpha)
  • large l1&l2 regularization strength will increase the impact of regularization, weights will be smaller (or more weights will be zero), model's capacity to fit train data will decrease, model is less overfitting, more generalized. However, if strength is too large, the model may be underfitting.

  • dropout regularization, randomly drop neurons during training process

    • can be applied to hidden layers and input layers
    • it prevents co-adaptation of neurons to reduce model overfitting and improving generalization
    • the dropout rate hyperparameter determines the probability of dropping out each neuron
    • large dropout rate means more neurons will be dropped out, this will lead to a stronger regularization. Model's capacity to fit train data will decrease, model is less overfitting, more generalized, However, if dropout rate is too large, model will lose excessive information, model may be underfitting.
  • Overall, l1 and l2 regularization aim to reduce overfitting by adding regularization terms to loss function to penalize large weights, where dropout regularization prevents overfitting by dropping out neurons during training process. They could be used individually or in combinationto improve model performance.

3.5.1 L1 regularization strength¶

In [23]:
config['parameters']['regularization']['values']=['l1'] 
config['parameters']['regularization_strength']['values']=[0.1,0.01,0.001,0.0001] 
config['parameters']['pool_type']['values']=['max'] 
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 4trv04st
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
wandb: Agent Starting Run: 5slxuh6l with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 0.0001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_103015-5slxuh6l
Syncing run visionary-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/5slxuh6l
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.021 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▄▁▆▄▄▄▄▄▃▃▃▅▄▄▆▄▄▄▇▃▅█▃▄▅▇▅▅▆▆▄▄▄▄▅▆▅▄▆▆
Test Accuracy▁▃▃▄▄▅▅▆▆▆▆▆▇▇▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Accuracy▁▃▄▄▄▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss█▅▄▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01546
Test Accuracy0.4581
Training Accuracy0.44264
Training Loss2.054

View run visionary-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/5slxuh6l
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_103015-5slxuh6l/logs
wandb: Agent Starting Run: rkt2u088 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 0.001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_130025-rkt2u088
Syncing run laced-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/rkt2u088
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▅▅▅▅▅▇▅▄▃█▅▅▄▄▄▃▁▃▄▄▆▆▇▂▄▆▄▇▅
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▅▅▅▅▅▇▅▄▃█▅▅▄▄▄▃▁▃▄▄▆▆▇▂▄▆▄▇▅
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-2e-05
Test Accuracy0.1
Training Accuracy0.09998
Training Loss2.32457

View run laced-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/rkt2u088
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_130025-rkt2u088/logs
wandb: Agent Starting Run: ivnft1y4 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 0.01
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_135430-ivnft1y4
Syncing run daily-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/ivnft1y4
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▆▃▇▆▃▅▄▁▄▃█▆▇▇▄▂▄▄█▄▅▇▃▆▄▄▁▆▃▅▅▇▆█▇▇▃▆▃▇
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▆▃▇▆▃▅▄▁▄▃█▆▇▇▄▂▄▄█▄▅▇▃▆▄▄▁▆▃▅▅▇▆█▇▇▃▆▃▇
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00268
Test Accuracy0.1
Training Accuracy0.10268
Training Loss4.50213

View run daily-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/ivnft1y4
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_135430-ivnft1y4/logs
wandb: Agent Starting Run: 0jumk0tk with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 0.1
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_153045-0jumk0tk
Syncing run celestial-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/0jumk0tk
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▅▅▇▆▃▁▇▅▇▅█▅▆▆▂▂▄▆▆▄▇▅▇▅▆
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▅▅▇▆▃▁▇▅▇▅█▅▆▆▂▂▄▆▆▄▇▅▇▅▆
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00102
Test Accuracy0.1
Training Accuracy0.10102
Training Loss222.14423

View run celestial-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/0jumk0tk
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_153045-0jumk0tk/logs
wandb: Agent Starting Run: zw9mefty with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.0001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_162138-zw9mefty
Syncing run serene-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/zw9mefty
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231078…

Run history:


Accuracy difference▂▁▂▂▃▂▂▁▂▃▂▃▃▄▆▃▃▄▄▄▅▅▆▅▇▆▇▆▇▆▆▇▇▆▆██▇██
Test Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Accuracy▁▂▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01426
Test Accuracy0.7823
Training Accuracy0.79656
Training Loss1.67367

View run serene-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/zw9mefty
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_162138-zw9mefty/logs
wandb: Agent Starting Run: 0ryyp8gx with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_185443-0ryyp8gx
Syncing run lyric-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/0ryyp8gx
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▄▁▂▃▃▁▂▃▃▃▃▇▄▃▂▃▂▅▆▄▄▄▅▅▄▅▄▆▆▆▆▆▆▆▆▆▇▇█▆
Test Accuracy▁▃▄▄▄▅▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Accuracy▁▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01102
Test Accuracy0.7604
Training Accuracy0.74938
Training Loss1.75064

View run lyric-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/0ryyp8gx
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_185443-0ryyp8gx/logs
wandb: Agent Starting Run: enxrwrs4 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.01
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_212318-enxrwrs4
Syncing run icy-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/enxrwrs4
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.230982…

Run history:


Accuracy difference▆▁▆█▇▇█▇▆▇█▆▆▆▆▆▆▆▆▆▆▆▇▆▇▅▆▆▆▆▆▆▇▅▆▇▆▅▆▇
Test Accuracy▁▃▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇█▇█▇█████▇███████
Training Accuracy▁▁▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇███████████████████
Training Loss█▆▄▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.01806
Test Accuracy0.3452
Training Accuracy0.32714
Training Loss2.20585

View run icy-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/enxrwrs4
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_212318-enxrwrs4/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: sgoz6wez with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.1
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230503_235149-sgoz6wez
Syncing run frosty-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/4trv04st
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/sgoz6wez
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference█▂▁▃▂▂▃▂▂▂▂▂▂▁▂▃▁▂▃▁▃▁▃▁
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy█▂▁▃▂▂▃▂▂▂▂▂▂▁▂▃▁▂▃▁▃▁▃▁
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0027
Test Accuracy0.1
Training Accuracy0.0973
Training Loss2.30309

View run frosty-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/sgoz6wez
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230503_235149-sgoz6wez/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [ ]:
# l1 regularization strength of 0.1,0.01,0.001,0.0001 seems too large, so try small values 0.0005,0.00001, 0.000001
config['parameters']['regularization_strength']['values']=[0.0005,0.00001, 0.000001] 
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
3.5.1.1 L1 result and interpretation¶
  • L1 regularization strength of 0.1, 0.01, 0.001, 0.0001, 0.0005 are very strong, models either fail to converge or have very low accuracy.
  • with L1 strength of 0.00001 and 0.000001, models have simliar accuracy as models without regularization.

  • I would train with more epochs to see a long term regularization effect with strength of 0.00001 and 0.000001

In [19]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/L1-regularization--Vmlldzo0NDg5Njcz -h 1024
3.5.1.2 further test¶
  • All three models have very similar accuracy in testset.
  • L1 0.00001 showed good regularization with 250 epochs, as the accuracy gap is around 0.05, where models without regularization or with L1 0.000001 have the gap of 0.9.
  • The model generalization is improved from moderate overfitting to slightly overfitting.
  • So I would like to combine L1 regulaization (with optimal strength 0.00001) and dropout to check if the model generalization will be even better.
In [ ]:
# train with 250 epochs to check how much effect will the regularization have
config["parameters"]["epochs"]["values"] = [250]
config['parameters']['regularization']['values']=['l1']
config['parameters']['regularization_strength']['values']=[0.00001,0.000001]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
In [20]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/L1-regularization-II--Vmlldzo0NDg5OTY5 -h 1024

3.5.2 L2 regularization strength¶

3.5.2.1 L2 result and interpretation¶
  • strength 0.0001 seems too weak, the model performance is similar to model without regularization
  • strength 0.1 and 0.01 are too strong, models performance is largely reduced
  • optimal strength is 0.001, where test accuracy is almost as good as before regularization, but accuracy gap is largely reduced.
  • I would use more epochs to better understand the regularization effect in long term
In [ ]:
config['parameters']['regularization']['values']=['l2'] 
config['parameters']['regularization_strength']['values']=[0.1,0.01,0.001,0.0001] 
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
In [21]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/L2-regularization--Vmlldzo0NDg5ODA0 -h 1024
3.5.2.2 further test¶
  • L2 strength 0.001, 0.0001 with 250 epochs

  • model with L2 0.0001 have almost same performance and overfitting as model without any regularization. This means this strength is too weak that the regularization effect is not helpful

  • with L2 0.001, model reaches similar high test accuracy as control model (no reg), and the model has no significant signs of overfitting (accuracy gap 0.03).
  • optimal L2 strength 0.001
In [22]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/L2-regularization-II--Vmlldzo0NDkwMDYy -h 1024

3.5.3 dropout rate¶

In [25]:
# dropout without l1&l2 regularization
config["parameters"]["dropout_rate"]["values"] = [0,0.1,0.3,0.5,0.7]
config["parameters"]["regularization"]["values"] = ['none']
config["parameters"]["regularization_strength"]["values"] = [0]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: fd1okk1w
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
wandb: Agent Starting Run: tx7fi110 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_065109-tx7fi110
Syncing run robust-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/tx7fi110
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.005 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.793764…

Run history:


Accuracy difference▄▁▂▂▂▂▁▁▁▃▁▂▂▂▃▄▄▅▂▂▃▅▃▄▄▅▅▅▅▅▆▆▆▆▇▇▇█▆▇
Test Accuracy▁▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇████████████
Training Accuracy▁▂▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▇▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01784
Test Accuracy0.7823
Training Accuracy0.80014
Training Loss1.6647

View run robust-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/tx7fi110
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_065109-tx7fi110/logs
wandb: Agent Starting Run: qnvpob18 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.1
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_091127-qnvpob18
Syncing run vibrant-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/qnvpob18
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.021 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▃▁▂▂▂▃▁▂▂▂▃▄▂▃▃▄▃▃▄▄▄▄▅▄▄▄▄▅▇▅▅▆█▅▆▆▆▇▇█
Test Accuracy▁▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇██▇███████████
Training Accuracy▁▂▃▃▄▄▄▄▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▆▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01398
Test Accuracy0.7641
Training Accuracy0.77808
Training Loss1.68604

View run vibrant-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/qnvpob18
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_091127-qnvpob18/logs
wandb: Agent Starting Run: s98c9ud0 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.3
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_114915-s98c9ud0
Syncing run skilled-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/s98c9ud0
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183277…

Run history:


Accuracy difference▁▂▃▃▄▄▃▆▄▄▅▆▃▅▃▄▄▅▄▄▅▄▅▅▆▅▄▄▇▅▆▆▆▆▇▇█▆▆▇
Test Accuracy▁▃▃▄▄▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇███████████
Training Accuracy▁▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Loss█▇▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00044
Test Accuracy0.7482
Training Accuracy0.74776
Training Loss1.71617

View run skilled-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/s98c9ud0
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_114915-s98c9ud0/logs
wandb: Agent Starting Run: qcc4f9gs with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.5
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_144732-qcc4f9gs
Syncing run royal-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/qcc4f9gs
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.016 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.783977…

Run history:


Accuracy difference▁▃▄▃▃▂▄▁▂▄▂▄▂▃▄▅▂▃▅▃▃▄▄▃▄▅▂▄▅▄▆▅▅▆▅▆▇▅▅█
Test Accuracy▁▂▃▃▄▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████
Training Accuracy▁▂▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████
Training Loss█▇▆▆▅▅▅▅▄▄▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00666
Test Accuracy0.7233
Training Accuracy0.71664
Training Loss1.74692

View run royal-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/qcc4f9gs
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_144732-qcc4f9gs/logs
wandb: Agent Starting Run: 3w203qyz with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.7
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.1 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230504_173404-3w203qyz
Syncing run resilient-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/fd1okk1w
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/3w203qyz
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231042…

Run history:


Accuracy difference▃▄▆▄▃▄▄▁▄▅▅▃▇▄▃▄█▆▄▇▄▅▄▄▅▆▆▃▅█▅▅▄▅▄▅▅▇▆▇
Test Accuracy▁▃▃▄▄▄▄▅▅▅▅▅▅▅▅▆▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇████████
Training Accuracy▁▃▃▄▄▄▄▄▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇███████
Training Loss█▇▆▆▆▅▅▅▅▅▄▄▄▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.0188
Test Accuracy0.7013
Training Accuracy0.6825
Training Loss1.78134

View run resilient-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/3w203qyz
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230504_173404-3w203qyz/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.5.3.1 dropout result and interpretation¶
  • we could see the dropout effect on model performance using different dropout rates
  • increase dropout rate, means larger proportion of neurons are dropped out, this will lead to smaller accuracy in both train and test set
  • however, more epochs is needed to find the suitable dropout rate
In [23]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/dropout-I--Vmlldzo0NDg5ODUx -h 1024
3.5.3.2 dropout with 150 epochs¶
In [33]:
# dropout regularization with 150 epochs
config["parameters"]["dropout_rate"]["values"] = [0.1,0.3,0.5,0.7]
config["parameters"]["regularization"]["values"] = ['none']
config["parameters"]["regularization_strength"]["values"] = [0]
config["parameters"]["epochs"]["values"] = [150]

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: wzmb66sy
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
wandb: Agent Starting Run: uxw8215b with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.1
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 150
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230507_054914-uxw8215b
Syncing run gallant-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/uxw8215b
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231031…

Run history:


Accuracy difference▁▂▃▂▃▂▂▂▃▂▃▃▃▄▃▄▄▅▅▅▅▅▅▅▆▆▆▆▇▆▆▇▇▆▇▇▇▇▇█
Test Accuracy▁▂▃▄▄▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Accuracy▁▂▃▃▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Loss█▇▆▆▅▅▅▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.05882
Test Accuracy0.8087
Training Accuracy0.86752
Training Loss1.5971

View run gallant-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/uxw8215b
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230507_054914-uxw8215b/logs
wandb: Agent Starting Run: 06477ds0 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.3
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 150
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230507_110232-06477ds0
Syncing run faithful-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/06477ds0
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231031…

Run history:


Accuracy difference▁▃▃▂▃▂▃▂▂▃▃▃▃▃▄▃▅▄▅▅▅▅▅▆▅▅▆▇▆▇▇▆▆▇▇▇████
Test Accuracy▁▂▂▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇████████████████
Training Accuracy▁▂▂▃▄▄▄▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Loss█▇▇▆▅▅▅▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.03388
Test Accuracy0.8044
Training Accuracy0.83828
Training Loss1.62527

View run faithful-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/06477ds0
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230507_110232-06477ds0/logs
wandb: Agent Starting Run: 4qd7nrds with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.5
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 150
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230507_161627-4qd7nrds
Syncing run zany-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/4qd7nrds
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231046…

Run history:


Accuracy difference▁▃▂▂▃▄▂▁▂▂▂▃▃▄▃▃▃▃▅▄▆▄▇▅▅▅▇▆▆▇▇▇▇▆▇▇████
Test Accuracy▁▂▃▃▄▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█▇█████████████
Training Accuracy▁▂▃▃▄▄▄▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇███████████
Training Loss█▇▆▆▅▅▅▄▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01874
Test Accuracy0.8018
Training Accuracy0.82054
Training Loss1.64333

View run zany-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/4qd7nrds
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230507_161627-4qd7nrds/logs
wandb: Agent Starting Run: pe62u3c0 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.7
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 150
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230507_205212-pe62u3c0
Syncing run restful-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/wzmb66sy
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/pe62u3c0
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183269…

Run history:


Accuracy difference▂▃▂▃▂▃▃▁▃▂▃▃▃▃▃▃▄▃▃▅▄▅▅▄▆▅▅▆▆▇█▅▆▇▇▇███▇
Test Accuracy▁▂▃▃▄▄▄▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇█████████
Training Accuracy▁▂▂▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████████
Training Loss█▇▇▆▆▅▅▅▅▄▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference-0.00144
Test Accuracy0.7924
Training Accuracy0.79096
Training Loss1.67254

View run restful-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/pe62u3c0
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230507_205212-pe62u3c0/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.5.3.3 result and interpretation¶
  • model with dropout rate = 0.1 has lower accuracy than model without any regularization, but the accuracy gap is almost same as without regularization. This indicates, the generalization of model with droput rate 0.1 is not improved.

  • with dropout rate = 0.7, the model accuracy gap is largely reduced, even below 0 after 150 epochs, this indicates, with so large rate of neurons dropout, models tend to be underfitting.

  • so i would select 0.3 and 0.5 for further tuning
In [24]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/dropout-II---Vmlldzo0NDg5ODg4 -h 1024

3.5.4 combine optimal L1 or L2 and dropout¶

In [36]:
# combine l1 and dropout regularization with 250 epochs
config["parameters"]["dropout_rate"]["values"] = [0.3,0.5]
config["parameters"]["regularization"]["values"] = ['l1']
config["parameters"]["regularization_strength"]["values"] = [0.00001]
config["parameters"]["epochs"]["values"] = [250]

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))

# combine l2 and dropout
config["parameters"]["dropout_rate"]["values"] = [0.3,0.5]
config["parameters"]["regularization"]["values"] = ['l2']
config["parameters"]["regularization_strength"]["values"] = [0.001]

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: 1ae5ri9l
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/1ae5ri9l
wandb: Agent Starting Run: 4njuffhd with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.3
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 250
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 1e-05
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230508_095126-4njuffhd
Syncing run genial-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/1ae5ri9l
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/1ae5ri9l
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/4njuffhd
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.420450…

Run history:


Accuracy difference▁▂▃▂▃▃▃▃▄▃▄▄▅▅▅▆▅▆▆▆▆▆▆▆█▆▇▆▆▇▇▇▇██▇▇▇█▇
Test Accuracy▁▃▄▄▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇███████████████████
Training Accuracy▁▃▃▄▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Loss█▆▅▅▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.02884
Test Accuracy0.8085
Training Accuracy0.83734
Training Loss1.72705

View run genial-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/4njuffhd
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230508_095126-4njuffhd/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: 9p748dov with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.5
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 250
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l1
wandb: 	regularization_strength: 1e-05
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230508_175743-9p748dov
Syncing run flowing-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/1ae5ri9l
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/1ae5ri9l
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/9p748dov
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.009 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.420450…

Run history:


Accuracy difference▁▃▂▃▂▃▃▄▂▃▄▃▄▄▄▄▄▄▄▅▅▅▅▅▅▅▅▆▆▅▆▆▇▆▆█▆▆▆▆
Test Accuracy▁▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇██████████████████
Training Accuracy▁▃▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Loss█▆▆▅▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.01268
Test Accuracy0.8046
Training Accuracy0.81728
Training Loss1.74872

View run flowing-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/9p748dov
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230508_175743-9p748dov/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
Create sweep with ID: l50fjgbw
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/l50fjgbw
wandb: Agent Starting Run: 51zkiagw with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.3
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 250
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230509_020930-51zkiagw
Syncing run good-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/l50fjgbw
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/l50fjgbw
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/51zkiagw
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.231092…

Run history:


Accuracy difference▁▄▂▃▂▃▃▂▂▃▃▄▄▄▄▄▄▅▅▄▆▇▆▇▆▆▇▆▆█▆▆█▆▆▇▇▆█▇
Test Accuracy▁▃▄▄▅▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█▇████████████████
Training Accuracy▁▃▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████████████
Training Loss█▆▅▅▅▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00788
Test Accuracy0.8165
Training Accuracy0.82438
Training Loss1.6846

View run good-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/51zkiagw
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230509_020930-51zkiagw/logs
wandb: Agent Starting Run: rpu564x4 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.5
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 250
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230509_103842-rpu564x4
Syncing run polished-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/l50fjgbw
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/l50fjgbw
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/rpu564x4
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.194620…

Run history:


Accuracy difference▁▂▁▂▁▂▁▁▃▁▂▃▂▃▄▃▄▄▄▅▄▃▄▄▆▅▄▅▄▅▄▅▅▅█▆▅▅▆▅
Test Accuracy▁▃▃▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇███▇███████████████
Training Accuracy▁▃▃▄▄▄▅▅▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇███████████████
Training Loss█▆▆▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.00672
Test Accuracy0.7983
Training Accuracy0.80502
Training Loss1.70378

View run polished-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/rpu564x4
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230509_103842-rpu564x4/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
3.5.4.1 interpretation & optimal regularization¶
  • all four models have largely improved the generalization.
  • l2 0.001 + dropout rate 0.3 has the highest test accuracy, and almost second smallest accuracy difference to the train set.
  • so I will use l2 regularization (strength = 0.001) combining with dropout (rate = 0.3) for my model
In [25]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/L1-dropout-L2-dropout--Vmlldzo0NDkwMTI5 -h 1024

3.5.5 final regularization effect: before VS after¶

  • in the report below, we could see a large difference between the models before and after regularization.
  • the regularization have a positive impact on the model's performance. Before regularization, the model reached a train accuracy of 0.9755 and a test accuracy of 0.858. after applying regularization, the train accuracy decreased to 0.8566, and the test accuracy slightly decreased to 0.8326. accuracy gap is reduced from 0.117 (moderate overfitting) to 0.024 (no significant overfitting).
  • this indicates that regularization has helped reducing overfitting and improved the model's generalization ability.

  • Overall, though the test accuracy has slightly decreased, the regularization technique has shown its effectiveness in reducing overfitting and improving the model's performance on unseen data. It is a common trade-off to sacrifice a bit of train accuracy to achieve better generalization and avoid overfitting.

In [51]:
# l2_0.001 + dropout 0.3
config['parameters']['regularization']['values'] = ['l2']
config['parameters']['regularization_strength']['values'] = [0.001]
config['parameters']['dropout_rate']['values'] = [0.3]
config['parameters']['epochs']['values'] = [500]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-regularization", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: kismk30b
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/kismk30b
wandb: Agent Starting Run: 2jj7rie6 with config:
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0.3
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 500
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: l2
wandb: 	regularization_strength: 0.001
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.2 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230510_151304-2jj7rie6
Syncing run autumn-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/kismk30b
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/sweeps/kismk30b
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/2jj7rie6
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.005 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.792972…

Run history:


Accuracy difference▁▁▁▁▂▂▃▂▃▃▄▄▄▄▄▇▅▅▅▆▆▆▆▆▆▇▆▆▇█▆▇▆█▆▇▇▇█▇
Test Accuracy▁▃▄▅▅▆▆▇▇▇▇▇▇▇█▇▇███████████████████████
Training Accuracy▁▃▄▄▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇█████████████████████
Training Loss█▆▅▅▄▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.02404
Test Accuracy0.8326
Training Accuracy0.85664
Training Loss1.65702

View run autumn-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-regularization/runs/2jj7rie6
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230510_151304-2jj7rie6/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [26]:
%wandb weiping-zhang/mc1-sgd-early-stop-regularization/reports/regularization-effect-before-VS-after--Vmlldzo0NDkwMTY4 -h 1024

3.6 Batch normalization (BatchNorm, BN) without regularization¶

  • in deep learning, batch normalization is often used to normalize the input in a convolutional or fully-connected layer.
  • for each batch, it will subtract the batch mean and divide by the batch standard deviation. This means, the impact of batch-to-batch variations will be reduced.
  • This will provide a more stable and consistent gradient signal to stabilize the learning process and ensure the network could learn more efficiently.
  • Here, I will apply batch normalization to both convolutional and fully-connected layers

3.6.1 re-form the network structure to include batch normalization¶

In [45]:
class Net(nn.Module):
    def __init__(self,config):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, config['out_1'], config['kernel_size'], stride = config['stride'], padding = config['padding']) 
        self.conv2 = nn.Conv2d(config['out_1'], config['out_2'], config['kernel_size'], stride = config['stride'], padding = config['padding']) 
        self.conv3 = nn.Conv2d(config['out_2'], config['out_3'], config['kernel_size'], stride = config['stride'], padding = config['padding'])
        self.batch_norm = config['batch_norm']
        
        if self.batch_norm:
            self.bn1 = nn.BatchNorm2d(config['out_1'])
            self.bn2 = nn.BatchNorm2d(config['out_2'])
            self.bn3 = nn.BatchNorm2d(config['out_3'])

        self.relu = nn.ReLU()
        if config['pool_type'] == 'max':
            self.pool = nn.MaxPool2d(2)
        elif config['pool_type'] == 'avg':
            self.pool = nn.AvgPool2d(2)
            
        # output size of each layer (including pooling) 
        self.size1 = math.floor((32 + 2 * config['padding'] - config['kernel_size']) / config['stride'] + 1) 
        self.size2 = math.floor(math.floor((self.size1 + 2 * config['padding'] - config['kernel_size']) / config['stride'] + 1)/2)
        self.size3 = math.floor(math.floor((self.size2 + 2 * config['padding'] - config['kernel_size']) / config['stride'] + 1)/2)
    
        self.fc1 = nn.Linear(config['out_3']*(self.size3)**2, 128)  
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(p=config['dropout_rate'])

    def forward(self, x):
        if self.batch_norm:
            x = self.relu(self.bn1(self.conv1(x)))
            x = self.pool(self.relu(self.bn2(self.conv2(x))))
            x = self.pool(self.relu(self.bn3(self.conv3(x))))
        else: 
            x = self.relu(self.conv1(x))
            x = self.pool(self.relu(self.conv2(x)))
            x = self.pool(self.relu(self.conv3(x)))
            
        x = x.view(x.size(0), -1)
        x = self.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x
In [6]:
# test with small dataset
config = {
    "method": "grid",
    "metric": {'name': 'Test Accuracy', 'goal': 'maximize'},
    "parameters": {
        "data_size":{"values":["small"]},
        "early_stop_epochs":{"values":[20]},
        "batch_norm":{"values":[True]},
        "out_1":{"values":[64]},
        "out_2":{"values":[128]},
        "out_3":{"values":[512]},
        "kernel_size":{"values":[4]},
        "stride":{"values":[1]},
        "padding":{"values":[1]},
        "pool_type":{"values":["max"]},
        "optimizer": {"values": ["sgd"]},
        "learning_rate": {"values": [0.01]},
        "batch_size": {"values": [32]},
        "momentum": {"values": [0]},    
        "epochs": {"values": [200]},    
        "dropout_rate": {"values": [0]},     # dropout = 0 -> drop out no neurons 
        "regularization": {"values": ["none"]},  # no regularization
        "regularization_strength": {"values": [0]}}}

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop-test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: aqid82t9
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test/sweeps/aqid82t9
wandb: Agent Starting Run: ozqea329 with config:
wandb: 	batch_norm: True
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.01667008746492987, max=1.0)…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_080538-ozqea329
Syncing run sleek-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test/sweeps/aqid82t9
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test/sweeps/aqid82t9
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test/runs/ozqea329
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▅▆▆████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▅▆▆████████████████████████████████████
Training Loss▇▆█▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.00452

View run sleek-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop-test/runs/ozqea329
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_080538-ozqea329/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [9]:
# apply batchnorm on normal dataset
config['parameters']['data_size']['values'] = ['normal']
config['parameters']['epochs']['values'] = [80]
sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: uzngj6mb
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/uzngj6mb
wandb: Agent Starting Run: zvsvn144 with config:
wandb: 	batch_norm: True
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 80
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: weiping-zhang. Use `wandb login --relogin` to force relogin
wandb version 0.15.3 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_001110-zvsvn144
Syncing run morning-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/uzngj6mb
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/uzngj6mb
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/runs/zvsvn144
Waiting for W&B process to finish... (success).

Run history:


Accuracy difference▄▁▃▃▃▄▃▄▄▅▄▄▄▆▆▅▅▅▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▆█▇▆▇
Test Accuracy▁▅▅▆▆▆▇▇▇▇▇▇▇▇▇▇▇███████████████████▇███
Training Accuracy▁▃▄▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇████████████
Training Loss█▆▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.07762
Test Accuracy0.8546
Training Accuracy0.93222
Training Loss0.19352

View run morning-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/runs/zvsvn144
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_001110-zvsvn144/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [10]:
# with 300 epochs
config['parameters']['epochs']['values'] = [300]

sweep_id = wandb.sweep(config, project="mc1-sgd-early-stop", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: v4tdfret
Sweep URL: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/v4tdfret
wandb: Agent Starting Run: lpp3wych with config:
wandb: 	batch_norm: True
wandb: 	batch_size: 32
wandb: 	data_size: normal
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 20
wandb: 	epochs: 300
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: sgd
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb version 0.15.3 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230521_130523-lpp3wych
Syncing run graceful-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/v4tdfret
View project at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop
View sweep at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/sweeps/v4tdfret
View run at https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/runs/lpp3wych
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.183075…

Run history:


Accuracy difference▁▂▂▃▄▄▄▅▅▅▆▅▆▇▇▆▇▇█▇█▇▇▇▇██▇█▇██████████
Test Accuracy▁▃▅▅▆▇▇▇▇▇▇▇▇▇▇███▇█▇███████████████████
Training Accuracy▁▂▄▅▅▅▆▆▆▇▇▇▇▇▇▇▇███████████████████████
Training Loss█▇▅▅▄▄▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.11168
Test Accuracy0.8835
Training Accuracy0.99518
Training Loss0.01383

View run graceful-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-sgd-early-stop/runs/lpp3wych
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230521_130523-lpp3wych/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.

3.6.2 interpretation¶

  • learning efficiency with normbatch is much higher than without normbatch
    • with normbatch, model is able to learn the input pattern and generalize to testset with both accuracies > 0.8 with 22 epochs
    • To reach the same accuracies, model without normbatch needs 80 epochs.
In [27]:
%wandb weiping-zhang/mc1-sgd-early-stop/reports/NormBatch--Vmlldzo0NDkwNDEz -h 1024

3.7 adam optimizer (without batchnorm, without regularization)¶

  • Adam optimizer (Adaptive Moment Estimation) is an adaptive optimization algorithm that adjusts the learning rate based on the historical gradients. It keeps track of the first and second moments of the gradients to estimate the adaptive learning rate for each parameter
  • first moment (mean or average of the gradients) helps the optimizer to move in the right direction. It keeps a moving average of the gradients to update the parameters effectively.

  • second moment (variance of the gradients) helps the optimizer to adjust the learning rate based on the scale of the gradients. It keeps a moving average of the squared gradients to normalize the learning rate.

  • They will help the optimizer to navigate through the optimization landscape efficiently and converge faster towards an optimal solution.

3.7.1 test on small dataset¶

In [7]:
# test with small dataset
config['parameters']['batch_norm']['values'] = [False]
config['parameters']['optimizer']['values'] = ['adam']
config['parameters']['data_size']['values'] = ['small']
config['parameters']['early_stop_epochs']['values'] = [50]
config['parameters']['epochs']['values'] = [200]
config['parameters']['learning_rate']['values'] = [0.0001,0.001,0.01,0.1]
config['parameters']['batch_size']['values'] = [16,32,64]
sweep_id = wandb.sweep(config, project="mc1-adam-early-stop-test", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
Create sweep with ID: cyrvyzoi
Sweep URL: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
wandb: Agent Starting Run: ugzhg27j with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016670136316679417, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_081159-ugzhg27j
Syncing run lively-sweep-1 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/ugzhg27j
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▁▁▅▅█████████▅█▅████▅█████▅████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▅▅█████████▅█▅████▅█████▅████████████
Training Loss█▄▆▃▂▁▁▁▁▁▁▁▁▁▃▁▃▁▁▁▁▅▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.00103

View run lively-sweep-1 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/ugzhg27j
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_081159-ugzhg27j/logs
wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: Agent Starting Run: 1lcly50n with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.01667073113300527, max=1.0)…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_081403-1lcly50n
Syncing run graceful-sweep-2 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/1lcly50n
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▃▅▅▅█▆▆▆▆▆▆▆███████▅▅▆▆██▅██▆█▆███▆▆█▆█
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▃▅▅▅█▆▆▆▆▆▆▆███████▅▅▆▆██▅██▆█▆███▆▆█▆█
Training Loss▁▂▁▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁▁▁▄▅▂▁▁▁█▁▁▁▁▂▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.0

View run graceful-sweep-2 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/1lcly50n
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_081403-1lcly50n/logs
wandb: Agent Starting Run: fx3anw3a with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 16
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.01666964531953757, max=1.0)…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_081552-fx3anw3a
Syncing run graceful-sweep-3 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/fx3anw3a
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▅▃▃▅▆▆▅▅▅▃██▆▅▅▆▆▆█▅██▆▆▆███▆▆▆██▅████▆
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▅▃▃▅▆▆▅▅▅▃██▆▅▅▆▆▆█▅██▆▆▆███▆▆▆██▅████▆
Training Loss▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.75
Test Accuracy0.0
Training Accuracy0.75
Training Loss549.39795

View run graceful-sweep-3 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/fx3anw3a
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_081552-fx3anw3a/logs
wandb: Agent Starting Run: fhpuayk8 with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016669730100935944, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_081730-fhpuayk8
Syncing run magic-sweep-4 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/fhpuayk8
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▅█▅████████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▅█▅████████████████████████████████████
Training Loss█▇▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.0

View run magic-sweep-4 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/fhpuayk8
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_081730-fhpuayk8/logs
wandb: Agent Starting Run: n8tnqyqw with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016668457114913812, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_081951-n8tnqyqw
Syncing run likely-sweep-5 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/n8tnqyqw
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.232617…

Run history:


Accuracy difference▁▁▁▁▁▅██▅▅██████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▅██▅▅██████████████████████████████
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.00015

View run likely-sweep-5 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/n8tnqyqw
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_081951-n8tnqyqw/logs
wandb: Agent Starting Run: uzaky3cr with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 32
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016670338332187385, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_082314-uzaky3cr
Syncing run silvery-sweep-6 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/uzaky3cr
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▃▁▁▃▃▃▃▆▆▁▃▃▆██▃▃███▆▆▃█▆▆▆▆▆▆▆▆▆▆▆▆█▆▃
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▃▁▁▃▃▃▃▆▆▁▃▃▆██▃▃███▆▆▃█▆▆▆▆▆▆▆▆▆▆▆▆█▆▃
Training Loss▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.5
Test Accuracy0.0
Training Accuracy0.5
Training Loss134.60635

View run silvery-sweep-6 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/uzaky3cr
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_082314-uzaky3cr/logs
wandb: Agent Starting Run: y4b80qz4 with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.001
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.0166697686809736, max=1.0))…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_082458-y4b80qz4
Syncing run vocal-sweep-7 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/y4b80qz4
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.005 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.232652…

Run history:


Accuracy difference▁▁█▅▅███████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁█▅▅███████████████████████████████████
Training Loss▃▃▂▂█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference1.0
Test Accuracy0.0
Training Accuracy1.0
Training Loss0.0

View run vocal-sweep-7 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/y4b80qz4
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_082458-y4b80qz4/logs
wandb: Agent Starting Run: dz4k0izw with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.01
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.016669313000359884, max=1.0…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_082816-dz4k0izw
Syncing run avid-sweep-8 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/dz4k0izw
Early stopping
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.004 MB of 0.004 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=1.0, max…

Run history:


Accuracy difference▁▁▁▁▁▆▆▃▆▆▆▆▃▆▆▃▃▃▆▆█▃▆▆▆▆▆▆▃▆▆▆▃▆▆▆▆▆▆▆
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy▁▁▁▁▁▆▆▃▆▆▆▆▃▆▆▃▃▃▆▆█▃▆▆▆▆▆▆▃▆▆▆▃▆▆▆▆▆▆▆
Training Loss▂▄▂▁▁▁▁▁▁▁▁▁▃▁▁▁▁▆▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.75
Test Accuracy0.0
Training Accuracy0.75
Training Loss0.47853

View run avid-sweep-8 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/dz4k0izw
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_082816-dz4k0izw/logs
wandb: Agent Starting Run: ar1iai40 with config:
wandb: 	batch_norm: False
wandb: 	batch_size: 64
wandb: 	data_size: small
wandb: 	dropout_rate: 0
wandb: 	early_stop_epochs: 50
wandb: 	epochs: 200
wandb: 	kernel_size: 4
wandb: 	learning_rate: 0.1
wandb: 	momentum: 0
wandb: 	optimizer: adam
wandb: 	out_1: 64
wandb: 	out_2: 128
wandb: 	out_3: 512
wandb: 	padding: 1
wandb: 	pool_type: max
wandb: 	regularization: none
wandb: 	regularization_strength: 0
wandb: 	stride: 1
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
VBox(children=(Label(value='Waiting for wandb.init()...\r'), FloatProgress(value=0.0166688163842385, max=1.0))…
Tracking run with wandb version 0.15.0
Run data is saved locally in /home/jovyan/work/persistent/wandb/run-20230520_083009-ar1iai40
Syncing run divine-sweep-9 to Weights & Biases (docs)
Sweep page: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View project at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test
View sweep at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/sweeps/cyrvyzoi
View run at https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/ar1iai40
Waiting for W&B process to finish... (success).
VBox(children=(Label(value='0.008 MB of 0.021 MB uploaded (0.000 MB deduped)\r'), FloatProgress(value=0.387822…

Run history:


Accuracy difference█▁██▁███████████████████████████████████
Test Accuracy▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
Training Accuracy█▁██▁███████████████████████████████████
Training Loss█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

Run summary:


Accuracy difference0.5
Test Accuracy0.0
Training Accuracy0.5
Training Loss1.04586

View run divine-sweep-9 at: https://wandb.ai/weiping-zhang/mc1-adam-early-stop-test/runs/ar1iai40
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20230520_083009-ar1iai40/logs
wandb: Sweep Agent: Waiting for job.
wandb: Sweep Agent: Exiting.
In [28]:
%wandb weiping-zhang/mc1-adam-early-stop-test/reports/test-by-small-dataset--Vmlldzo0NDg0NDIz -h 1024

3.7.2 tune learning rate and batch size on normal dataset¶

when using the Adam optimizer, it is important to tune the learning rate and batch size. Although Adam adapts the learning rate automatically, finding an appropriate initial learning rate and batch size can still significantly impact the training process.

In [ ]:
config['parameters']['data_size']['values'] = ['normal']
config['parameters']['early_stop_epochs']['values'] = [20]
config['parameters']['epochs']['values'] = [50]
config['parameters']['learning_rate']['values'] = [0.00001,0.0001,0.001,0.01]
config['parameters']['batch_size']['values'] = [16,32,64,128]
sweep_id = wandb.sweep(config, project="mc1-adam-early-stop", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
3.7.2.1 optimal learning rate & batch size¶
  • optimal learning rate is 0.0001, batch size 64 as the model has very high performance and short computational time.
In [29]:
%wandb weiping-zhang/mc1-adam-early-stop/reports/Adam-tune-learning-rate-batch-size--Vmlldzo0NDg0MzQx -h 1024
3.7.3 train data with 300 epochs using optimal learning rate & batch size¶
In [ ]:
# 300 epochs with optimal learning rate 0.0001 batch size 64
config['parameters']['early_stop_epochs']['values'] = [20]
config['parameters']['epochs']['values'] = [200]
config['parameters']['learning_rate']['values'] = [0.0001]
config['parameters']['batch_size']['values'] = [64]
sweep_id = wandb.sweep(config, project="mc1-adam-early-stop", entity="weiping-zhang")
wandb.agent(sweep_id, function=lambda: cnn_train_evaluation(config))
In [30]:
%wandb weiping-zhang/mc1-adam-early-stop/reports/Adam-vs-SGD--Vmlldzo0NDkwNDgy -h 1024
3.7.4 interpretation adam vs sgd¶
  • adam optimizer is more efficient than sgd optimizer.
    • with adam, model needs only 16 epochs to reach accuracy > 0.8 of both train and testset
    • with sgd, 89 epochs is needed to reach the arruracy > 0.8 on both dataset
  • adam optimizer could largely speed up the training process.

4 Summary¶

  • The best model with sgd optimizer is a

    • 3 convolutional layers + 2 fully connected layer model
    • with 64,128,512 filters
    • kernel size = 3, stride = 1, padding = 0, max pooling
    • dropout rate = 0.3
    • l2 regularization strength 0.001
    • learning rate 0.0001
    • batch size 32
  • Adam optimizer is efficient due to its adaptive learning rate to each parameter

  • with Normbatch, model could stablize learning process and improve the learning efficiency by reducing the impact of batch-to-batch variations
  • SGD optimizer needs fine hyperparameter tuning